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1.  INTRODUCTION 


Decision  analysis  is  Che  systematic  application  of  decision  theory 
to  problems  of  choice  among  alternatives.  As  such  it  has  developed  over 
the  last  two  decades  into  a  mature  and  powerful  approach.  It  can  be 
applied  to  a  large  nunber  of  problems,  and  there  are  many  well  developed 
techniques  for  assisting  the  decision  maker  in  choosing  among  competing 
alternatives.  For  examples  see  Barclay  et  al[8],  Holloway [34] , 
Raiffa[64j. 

In  practice,  however,  the  decision  maker  may  expend  as  or  more  much 
effort  in  specifying  the  alternatives  (option  generation)  as  he  does  in 
assessing  their  value  (choice  resolution).  The  vital  role  that  option 
generation  plays  in  the  overall  success  of  the  decision  making  process 
is  obvious.  If  the  "best"  alternative  is  not  in  the  choice  set,  then  it 
can  never  be  selected. 

Unfortunately,  human  decision  makers  are  less-than-perfect 
gatherers  and  processors  of  information,  are  subject  to  a  variety  of 
cognitive  biases,  and  display  personal  idiosyncracies  in  their  decision 
making.  It  is  not  surprising  then  that  when  faced  with  difficult 
problems  ,  and  especially  in  times  of  stress,  they  exhibit  demonstrably 
suboptimal  option  generation  and  choice  resolution  behavior. 

Our  primary  task  on  this  research  contract  has  been  to  focus  on 
option  generation  in  novel  and  stressful  situations,  and  investigate 
whether  tools  can  be  developed  that  would  help  the  decision  maker 
generate  better  sets  of  options.  In  our  research  we  have  first  concerned 
ourselves  with  the  reasons  why  decision  makers  perform  so  poorly,  and 
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Chen  used  that  Knowledge  to  define  what  we  have  called  an  Option 
Prompting  Kny j pmonf .  We  have  performed  a  thorough  search  of  the 
appropriate  literature  and  found  few  reports  on  the  specific  problem  of 
option  generation  in  decision  making.  There  is,  however,  a  significant 
body  of  psychology  literature  that  deals  with  creativity  and  problem 
solving,  and  we  have  used  some  significant  ideas  from  these  sources  in 
developing  our  approach. 

Much  of  our  early  effort  was  devoted  to  developing  the  concepts  of 
problem  structuring  tools,  but  our  later  work  has  attempted  to  define  a 
broader  generic  view  of  the  decision  process  so  that  option  generation 
can  be  understood  in  its  proper  context.  The  resulting  more 
comprehensive  characterization  of  the  decision  process  has  led  us  to  the 
conclusion  that  the  option  generation  process  depends  on  acquiring 
problem  relevant  data  and  domain  knowledge,  problem  structuring,  stating 
option  ,  evaluating  and  refining  options,  and  using  procedures  that  are 
likely  to  evoke  new  ideas.  Decision  makers  may  perform  these  activities 
iteratively  or  in  a  more  random  fashion. 

Recognizing  the  importance  of  formal  testing  of  decision  aids,  we 
have  performed  two  separate  experiments  that  were  designed  to  explore 
and  evaluate  our  ideas.  We  have  used  the  results  so  obtained  to  guide 
our  thinking  about  what  constitutes  an  appropriate  decision  making 
environment,  and  the  types  of  tools  that  should  be  included  in  that 
environment  to  enhance  the  option  generation  process. 

The  Air  Force  is  willing  to  invest  relatively  large  amounts  of 
money,  time  and  effort  in  the  explicit  training  of  pilots,  other 
operator  personnel,  and  technicians;  but  little  for  explicit  training  of 


Generals  and  staff  officers  in  one  of  their  most  important  activities: 
command  decision  making.  Providing  decision  support  systems  such  as 
Option  Prompting  Environments  would  provide  mechanisms  for  both  staff 
colleges  and  operational  staffs  to  focus  efforts  in  this  area.  In 
particular,  it  may  encourage  them  to  focus  on  making  "better"  decisions 
(i.e.,  generating  and  selecting  a  "good"  alternative)  instead  of 
immediately  developing  a  plan  for  an  "OK"  alternative. 

The  remainder  of  this  technical  report  is  organized  into  six 
chapters  and  a  number  of  appendices.  Chapter  2  contains  a  discussion  of 
the  literature  we  have  surveyed.  There  is  a  large  amount  of  material 
that  deals  with  the  general  question  of  decision  making  and  problem 
solving,  so  we  have  attempted  to  categorize  the  literature  into  several 
useful  domains.  We  have  restricted  our  review  in  this  chapter  to  those 
papers  that  we  have  designated  as  being  significant  to  an  understanding 
of  the  option  generation  problem  and  our  view  of  it .  A  fully  annotated 
bibliography  begins  on  page  98  of  this  report. 

In  Chapter  3  we  develop  our  concept  of  an  option  prompting 
environment.  We  start  by  examining  the  overall  decision  making  process 
and  describe  several  models  that  help  us  understand  why  decision  making 
can  be  less  than  optimal .  We  then  use  some  recent  ideas  from  the 
psychology  of  creativity  to  define  the  properties  that  an  option 
prompting  environment  should  have.  The  role  that  cognitive  biases  and 
styles  play  in  this  environment  is  then  examined.  Central  to  this 
environment  is  the  provision  of  a  set  of  tools  that  the  decision  maker 
can  draw  upon.  We  define  some  appropriate  classes  of  tools  and  rules  for 
their  use.  To  conclude  the  chapter  we  describe  an  outline  for  a 
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computer  system  architecture  that  would  support  an  operational 
definition  of  our  envirooaent.  Seme  mathematical  interpretations  of  the 
issues  in  option  generation  are  given  in  Appendix  B. 

In  Chapter  4  we  consider  tools  for  helping  the  decision  maker 
structure  the  decision  problem  and  arrive  at  priorities  among  decision 
factors.  A  significant  part  of  our  our  effort  has  been  in  developing  and 
evaluating  a  computer  implementation  of  the  Analytic  Hierarchy  Process 
which  we  believe  to  be  a  particularly  useful  technique.  We  describe  thi6 
methodology,  review  is  effectiveness  and  suggest  areas  for  further 
development.  A  more  detailed,  and  mathematical,  description  of  AHP  is 
given  in  Appendix  C. 

Our  first  experiment  is  described  in  Chapter  5.  This  was  designed 
to  test  the  concept  of  AHP  as  a  structuring  tool,  and  we  report  on  the 
procedures  used  and  the  lessons  learned.  For  our  second  experiment, 
described  in  Chapter  6,  we  chose  to  re-test  our  AHP  software  together 
with  a  simple  version  of  the  option  prompting  environment.  We  also 
performed  a  series  of  secondary  experiments  to  try  and  correlate 
performance  in  the  main  experiment  with  performance  on  various  cognitive 
style  tests.  The  stimulus  material  used  in  the  second  experiment  is 
included  in  Appendix  D  for  reference. 

In  the  final  chapter  we  survey  the  status  of  the  project  and 
suggest  areas  for  further  research  and  development.  We  also  attempt  to 
assess  the  likely  impact  of  an  operational  option  prompting  environment 
on  Air  Force  decision  making. 

A  mapping  between  the  four  tasks  defined  in  the  original  Research 
and  Technology  Work  Statement  and  the  report  is  shown  in  Table  1.1. 
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Table  1.1.  Mapping  Between  Tasks  in  SOW 
and  Effort  Described  in  Report 


TASK 

REPORT  SECTIONS 

•  Analyze  and  document 

1 i terature 

•  Chapter  2;  Appendix  A 

•  Define  techniques  for 
option  generation 

•  Chapters  3,  4;  Appendix  B,  C 

•  Provide  and  define 

experiments 

•  Chapters  5,  6;  Appendix  D 

•  Analyze  impact  on  AF 
decision  making 


•  Chapter  7 


2.  LITERATURE  SURVEY 


There  is  a  substantial  amount  of  work  reported  in  the  literature 
that  has  potential  relevance  to  the  problem  of  option  generation.  We 
have  attempted  to  do  three  things  in  our  survey:  compile  an  extensive 
bibliography,  partition  the  entries  into  a  variety  of  categories,  and 
highlight  the  key  papers.  This  chapter  describes  the  major  categories  we 
have  chosen  and  reviews  the  papers  we  believe  to  be  most  important  in 
understanding  the  option  generation  problem.  A  more  detailed  survey  is 
given  in  the  annotated  bibliography  and  in  Appendix  A. 

2..1. .  A  Categorization  of  the  Literature 

In  our  search  for  relevant  literature  we  have  found  that  many 
disciplines  are  concerned  with  understanding  how  solutions  to  problems 
are  generated.  Perhaps  this  is  not  surprising,  since  a  great  deal  of 
human  effort  is  devoted  to  exactly  this  task.  Indeed, 


For  the  purposes  of  our  survey  we  have  made  a  distinction  between 
those  disciplines  that  concern  themselves  primarily  with  the 
psychological  processes  of  decision  making  and  those  disciplines  that 
address  the  situational  aspects  of  decision  making.  In  the  former, 
"internal",  category  we  haved  placed  psychology,  cognitive  science  and 
artificial  intelligence.  In  the  latter,  "external",  category  we  have 
placed  decision  analysis,  management  science  and  systems  engineering.  In 


practice  this  partition  is  fuzzy  since  many  studies  are  concerned  with 
both  the  internal  and  external  issues  in  decision  making.  Nonetheless, 
all  the  literature  that  ve  have  considered  has  as  its  goal  either  the 
development  of  models  of  decision  making  and  problem  solving,  or  the 
definition  of  appropriate  procedures  for  improving  the  quality  of 
decisions. 

Another  useful  way  of  dividing  the  literature  is  into  various 
subject  categories.  We  have  defined  five  categories  that  reflect  the 
range  of  issues  that  arise  in  option  generation.  First,  we  have  looked 
at  studies  in  behavioral  decision  theory.  These  concern  themselves  with 
psychological  aspects  of  judgement  and  choice,  and  are  particularly 
concerned  with  patterns  of  behavior  for  coping  with  uncertainty.  The 
literature  on  cognitive  biases  and  styles  is  included  in  this  category. 

Our  second  subject  category  is  creativity,  and  we  included  studies 
designed  to  define,  measure  and  improve  it.  This  category  is  very  broad, 
ranging  from  formal  psychological  investigations  to  "popular" 
suggestions  for  improving  personal  problem  solving  behavior. 

The  third  subject  category  is  decision  support  systems  (DSS). 
Again,  this  is  a  rather  broad  classification  devised  to  encompass  any 
studies  that  discuss  the  design  and  implementation  of  tools,  or  systems 
of  tools,  for  helping  decision  makers.  The  tools  may  be  elaborate,  an 
extensive  management  information  system  (MIS)  for  example,  or  simple, 
such  as  straightforward  checklists  for  routine  tasks. 

The  fourth  subject  category  is  concerned  with  methodological 
questions.  In  particular,  we  include  studies  that  look  at  the  design  of 
experiments  and  appropriate  measures  of  performance.  The  final  category 
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is  problem  solving.  This  ia  least  clearly  defined  and  is  something  of  a 
catch-all.  In  it  ve  have  placed  studies  in  organization  theory, 
planning,  formal  reasoning  and  decision  analysis. 

A  further  possible  categorization  is  by  study  type,  and  ve  have 
distinguished  four  kinds.  First,  there  are  studies  that  are  primarily 
concerned  with  experimental  verification  of  hypotheses  about  decision 
making  behavior.  Second,  there  are  studies  whose  main  concern  is  with 
the  detailed  implementation  of  decision  aids,  be  they  computerized 
decision  systems  or  formal  procedures.  Third,  there  is  a  broad  class  of 
studies  which  we  consider  to  be  of  a  theoretical  nature.  These  include 
work  on  mathematical  theories  of  preference  and  choice,  presentations  of 
psychological  theories  of  problem  solving,  and  philosophical  enquiries 
into  the  foundations  of  human  reasoning.  Finally,  several  authors  have 
attempted  reviews  and  surveys  of  the  various  literatures  and  we  include 
these  in  our  miscellany  category. 

Obviously,  there  are  other  ways  in  which  this  partitioning  could 
have  been  performed.  However,  we  believe  that  our  approach  is  adequate 
for  introducing  the  literature  and  avoids  the  fragmentation  that  a  more 
precise  categorization  would  give.  The  scope  of  our  search  has  been 
restricted  to  English  language  publications,  but  otherwise  we  have 
attempted  as  broad  a  view  as  possible.  No  doubt  we  have  overlooked  some 
work  that  is  relevant,  but  by  including  in  our  bibliography  several 
review  and  survey  articles,  we  hope  that  everything  significant  has  been 
referenced  at  least  indirectly. 
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As  we  develop  our  view  of  the  Option  Generation  problem  in 
subsequent  chapters  we  will  refer  explicitly  to  some  important  work  that 
has  influenced  our  thinking.  The  purpose  of  this  section  is,  therefore, 
to  highlight  studies  that  provide  insight  into  the  multiplicity  of 
issues  that  impinge  on  Option  Generation. 

There  has  been  much  interest  in  behavioral  decision  making  and  two 
excellent  extensive  reviews  have  appeared.  Slovic,  Fischhoff  and 
Lichtenstein  [71]  concentrate  on  heuristics  and  biases  referencing  over 
300  publications;  Einhorn  and  Hogarth  [19]  discuss  the  question  of 
normative  versus  descriptive  approaches  to  human  decision  making  and 
reference  over  150  publications.  More  recently,  a  collection  of  papers 
has  appeared  edited  by  Kahnemann,  Slovic  and  Tversky  [45]  which  includes 
many  of  the  key  publications  on  this  topic. 

Perhaps  the  main  result  of  all  this  effort  has  been  the  recognition 
that  man  is  a  less-than-optimal  gatherer  and  processor  of  information. 
He  exhibits  a  wide  range  of  systematic  biases  in  all  stages  of  decision 
making  and  often  displays  personal  idiosyncacies  thus  impairing  his 
ability  to  make  fully  considered  decisions. 

While  much  experimental  evidence  has  been  gathered,  a  comprehensive 
theory  of  biases  that  would  allow  us  to  predict  when  they  will  operate 
has  yet  to  appear.  Indeed,  there  is  a  debate  over  the  validity  of  the 
effects  observed,  with  many  questions  about  the  confidence  we  should 
have  in  translating  the  results  of  the  laboratory  studies  to  "real- 
world"  decision  problems. 


There  seem  to  have  been  few  efforts  to  study  the  option  generation 
process  directly  with  the  work  of  Gettys  et  al  and  Pits  et  al  being  the 
most  prominent.  In  [23]  Gettys  et  al  report  on  a  series  of  experiments 
designed  to  test  the  adequacy  of  option  generation  performance.  This 
work  is  notable  for  its  development  of  a  methodology  for  measuring 
performance.  The  main  finding  was  that  subjects  failed  to  generate 
important  options  and  that  this  was  often  costly.  In  [25]  they  attempt 
to  construct  a  model  of  the  way  in  which  decision  makers  generate 
hypotheses  about  data.  The  model  is  based  on  Bayes  Theorem  with  the  idea 
that  new  hypotheses  are  only  generated,  or  retrieved  from  memory,  when 
the  currently  active  set  has  low  "plausibility".  Their  experimental  data 
seems  to  provide  some  support  for  the  model.  A  general  overview  of  this 
body  of  work  is  given  in  [24]. 

In  [60]  Pitz  et  al  examined  reasons  why  decision  makers  omit 
certain  decision  alternatives.  In  performing  their  experimental  work 
they  discovered  that  focusing  on  objectives  was  an  effective  way  of 
generating  choices.  Furthermore,  considering  the  objectives  one  at  a 
time  was  observed  to  be  better  than  considering  then  all  at  once.  In 
[61]  they  extend  this  work  and  conclude  that  a  scenario  script  (or 
template)  approach  to  problem  structuring  is  the  most  effective 
procedure  since  this  matches  human  tendencies  to  think  in  terms  of 
action-outcome  scenarios. 

Human  creativity  is  a  subject  that  has  been  widely  studied  both  by 
the  academic  community  and  by  a  variety  of  "lay"  practitioners.  We  have 
found  the  theoretical  work  of  Amabile  [4], [5]  to  be  particularly 
insightful.  Her  view  of  creativity  as  being  dependent  on  domain 
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knowledge,  motivation  and  creativity  skills  helps  understand  some  of  the 
very  widely  differing  results  of  creativity  research.  An  older,  but 
very  comprehensive,  discussion  of  the  role  of  creativity  in  human 
endeavour  is  given  in  Koestler [47 ] ,  and  at  a  less  theoretical  level,  the 
work  of  Adams  [2]  is  an  excellent  example  of  the  design  of  procedures 
which  can  help  overcome  certain  kinds  of  mental  blocks  (biases). 

There  have  been  several  attempts  to  build  computerized  aids  for 
decision  making  where  the  emphasis  has  been  on  problem  structuring.  See 
for  example  the  work  reported  by  Gulick[27]  and  by  Merkhoffer  et  al 
[53].  While  most  of  these  use  conventional  decision  trees,  the  work  of 
Pearl  et  al  [59]  is  substantially  different.  Their  basic  premise  is  that 
in  many,  if  not  all,  real-world  applications  the  decision  maker  does  not 
perceive  the  problem  as  a  time  sequence  of  decision  alternatives  and 
event  outcomes.  Rather,  he  sees  a  static  network  of  influences 
surrounding  issues  and  factors.  Thus  when  the  decision  make  confronts  a 
complex  problem  he  does  not  think  in  terms  of  the  alternatives  available 
to  him,  but  in  terms  of  his  desires  and  concerns.  This  leads  the 
authors  to  consider  an  alternative  structure  for  representing  the 
decision  maker's  knowledge  as  a  hierarchy  of  goals  and  sub-goals  with  a 
complex  form  of  sensitivity  analysis  used  to  guide  the  search  through 
the  goal  network.  The  authors  claim  that  the  goal  directed  approach  is 
superior  in  both  clarity  and  purposefulness  and  that  the  explicit 
mention  of  objectives  helps  the  decision  maker  evoke  unconventional 
alternatives  capable  of  realizing  those  objectives. 


Finally,  Wohl  [85]  provides  an  excellent  overview  of  the  issues  and 
decision  requirements  for  Tactical  Air  Force  Command  and  Control.  The 


focus  is  on  the  requirements  of  the  human  decision  maker  in  a  variety  of 
contexts  (both  preflight  and  inflight).  He  shows  how  decision  aids  can 
be  assessed  in  terms  of  a  behavioral  model  that  distinguishes  between 
stimulus  (data),  hypothesis  (perception  alternatives),  options  (response 
alternatives)  and  response  (actions). 

We  consider  each  of  the  works  mentioned  above  to  be  a  good 
introduction  to  their  respective  views  of  human  decision  making,  and 
they  provide  a  convenient  base  for  exploring  the  issues  in  more  detail. 
Appendix  A  contains  our  own  annotated  bibliography. 


3.  AN  OPTION  PROMPTING  ENVIRONMENT 


In  this  chapter  we  will  present  our  view  of  the  Option  Generation 
problem  and  discuss  a  basic  construct  that  we  believe  provides  the 
foundations  for  a  comprehensive  solution.  We  have  attempted  to  construct 
only  the  theoretical  skeleton  of  this  solution,  concerning  ourselves 
with  structuring  and  exploring  the  validity  of  our  primary  concepts 
rather  than  with  questions  of  how  the  solution  might  be  implemented. 

Our  basic  insight  is  that  the  decision  maker  needs  an  Option 
Prompting  Environment  rather  than  one  specific  option  generation  aid. 
We  will  show  how  we  have  been  led  to  this  view  by  first  discussing  the 
relationships  between  option  generation  and  the  rest  of  the  decision 
making  process,  and  then  by  considering  the  reasons  for  poor  performance 
on  the  option  generation  task.  We  define  the  characteristics  of  the 
environment  by  drawing  on  work  in  the  psychology  of  problem  solving, 
and,  in  particular,  consider  the  role  of  decision  maker  creativity. 
Cognitive  biases  and  styles  have  concerned  many  workers  in  decision 
making,  and  we  review  this  work  in  relation  to  options  generation  and 
our  solution.  A  feature  of  our  environment  is  that  the  decision  maker 
is  given  access  to  a  variety  of  potential  aids;  so  we  discuss  the 
concept  of  a  "toolbox"  and  the  rules  for  its  use.  Finally,  we  outline 
an  architecture  for  a  computerized  version  of  our  environment  and 
discuss  its  main  components,  showing  how  these  reflect  the  concerns 
raised  in  the  earlier  parts  of  the  chapter. 
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3_.  1_.  The  Option  Generation  Problem 


Stated  simply,  when  the  decision  maker  is  engaged  in  the  task  of 
option  generation  he  is  attempting  to  define  a  set  of  choices,  or 
action,  alternatives  to  which  he  can  apply  some  appropriate  normative 
technique  for  selecting  the  best.  We  feel  that  option  generation  is  a 
vital,  and  yet  neglected,  part  of  the  decision  making  process.  This  is 
somewhat  surprising  since  one  could  argue  that  the  value  of  decision 
analysis  is  primarily  in  the  structuring  it  gives  to  such  open  problems. 

"Practiced  decision  analysts  ...  report  that  a  major  part  of 
many  studies  is  the  specification  of  the  set  of  alternative 
courses  of  action." 

Watson  and  Brown[79] 

We  will  make  a  distinction  between  what  we  term  "closed"  decision 
problems,  such  as  buying  a  new  car,  and  "open"  ones,  such  a  rescuing  the 
hostages  from  the  embassy  in  Tehran.  A  closed  problem  is  one  in  which 
the  option  generation  task  is  straightforward,  either  because  the 
problem  is  routine  or  because  the  number  and  type  of  options  is  defined 
by  the  problem.  An  open  problem  on  the  other  hand  is  characterized  by 
being  novel  and  by  having  no  clear  a  priori  bounds  on  the  possible 
solution  set.  Furthermore,  the  "openness"  itself  often  induces  stress  in 
the  decision  maker  causing  even  further  deterioration  in  his  ability  to 
think  rationally  about  the  problem. 

Our  efforts  on  this  contract  have  been  directed  towards  option 
generation  for  open  problems.  Central  to  our  view  is  the  assumption 
that  the  actual  act  of  option  generation  cannot  be  automated.  It  is 
essentially  a  creative  activity  that  can  only  be  performed  by  the 


decision  maker.  All  we  can  hope  Co  do  is  enhance  Chis  creativity.  We 
will  return  to  this  point  later  and  for  now  concentrate  on  the  overall 
decision  making  process  and  discuss  some  reasons  why  decision  making  is 
often  less  than  perfect. 


3.2.  The  Process  of  Decision  Making 


Figure  3.1  shows  a  model  of  the  various  stages  of  decision  making. 
This  breakdown  into  six  distinct  phases  is  by  no  means  unique,  and  other 
authors  have  presented  similar  structures.  See  for  example  Hogarth[33]. 
The  purpose  of  this  model  is  to  show  that  tne  decision  maker  has  to 
perform  several  different  functions  as  he  moves  frcm  recognizing  that  he 
has  a  problem,  to  implementing  his  preferred  choice. 

The  most  important  features  of  this  model  are  the  feedforward  and 
feedback  loops.  Their  purpose  is  to  emphasize  that  decision  making  is 
not  a  linear  progression  from  one  stage  to  another.  Rather,  it  is 
essentially  iterative,  with  the  decision  maker  moving  as  required 
between  the  elements  we  have  shown,  with  the  movement  being  regular  or 
seemingly  random  To  illustrate  the  concept,  suppose  the  decision  maker 
realizes  that  the  problem  is  one  that  he  has  Been  and  successfully  dealt 
with  before.  In  this  case,  he  can  dispense  with  a  detailed  problem 
structuring  effort,  does  not  need  to  create  possible  solutions  or  assess 
their  relative  merits,  and  can  proceed  immediately  to  an  implementation 
of  the  solution.  This  is  indicated  in  our  model  by  the  feedforward  loop 
from  problem  recognition  to  implementation  and  review.  Such  “stimulus- 
response"  decision  making  has  been  of  little  interest  to  us  in  this 
contract;  it  corresponds  to  typical  behavior  in  what  we  have  called 
closed  problems. 


Much  more  challenging  is  Co  describe  typical  behavior  in  an  open 
problem.  In  such  cases,  Che  decision  maker  will  engage  in  activities  in 
each  of  the  six  blocks  in  our  model.  Faced  with  a  novel  problem,  the 
decision  maker  will  probably  want,  and  gather,  as  much  information  as 
possible  about  the  specific  decision  situation.  He  will  use  this 
information,  together  with  his  own  domain  knowledge,  to  begin 
structuring  the  problem.  This  somewhat  vague  concept  involves  a 
determination  of  the  major  factors  that  the  decision  maker  believes 
influence  the  problem.  These  might  include  the  decision  maker's  goals, 
the  goals  of  the  other  actors,  the  constraints  imposed  by  the 
environment  and  the  resources  that  the  decision  maker  can  bring  to  bear 
on  the  problem  either  directly  or  indirectly.  As  his  understanding  of 
the  problem  increases,  the  decision  maker  will  begin  to  think  about 
possible  solutions  in  addition  to  any  he  may  have  thought  of  when  he 
first  became  aware  of  the  decision  situation.  Depending  upon  his 
personal  style,  he  may  attempt  to  evaluate  the  feasibility  of  the 
options  as  soon  as  he  thinks  of  them,  or  he  may  generate  as  many  as 
possible  before  proceeding  to  the  analysis  of  specific  options.  It  is 
quite  conceivable  that  as  options  are  generated  and  evaluated  the 
decision  maker's  view  of  the  problem  changes.  He  might  engage  in  a 
cycle  of  structuring,  option  generation  and  evaluation  until  he  is 
confident  that  his  understanding  of  the  problem  is  complete.  Only  at 
this  point  will  he  proceed,  first  to  a  formal  analysis  of  the  options 
and  then  to  implementation  of  his  choice.  Even  then,  though,  the 
overall  cycle  may  not  be  completed.  In  many  instances,  the  outcome  of 
decision  making  is  a  plan  of  action,  and  it  is  an  important  part  of. the 
decision  maker's  function  to  monitor  the  execution  of  the  plan.  While 
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doing  this  he  may  become  aware  of  new  information  about  the  situation 
which  would  lead  him  to  revise  his  initial  decision.  Depending  upon  the 
magnitude  of  the  deviation  from  the  outcomes  originally  conceived,  this 
revision  may  take  the  form  of  a  drastic  restructuring  or  just  a  simple 
modification. 

Our  model  can  thus  provide  a  descriptive  account  of  a  set  of 
typical  decision  behaviors.  Its  main  purpose,  however,  is  to  show  that 
options  generation  is  intimately  connected  with  the  other  parts  of  the 
decision  making  process.  It  makes  little  or  no  sense,  therefore,  to 
treat  the  option  generation  problem  in  isolation.  If  we  do,  then  we 
are  ignoring  the  essential  iterative  and  interconnected  nature  of  the 
decision  making  process.  It  is  important  to  understand,  and  take 
account  of,  the  various  functions  the  decision  maker  has  to  perform,  and 
we  have  to  examine  how  these  affect  the  central  question  of  options 
generation. 

Having  described  the  nature  of  the  overall  decision  making  process, 
we  need  to  ask  why  it  is  that  decision  makers  often  find  it  difficult  to 
generate  options  for  open  problems.  Unfortunately,  as  our  literature 
survey  has  shown,  there  are  no  truly  satisfactory  explanations  and  we 
can  conclude  that  there  are  a  variety  of  reasons  which  are  more  or  less 
relevant  for  different  decision  makers  in  different  contexts. 

Our  approach  to  this  question  is  to  construct  a  model  of  "tasks" 
that  parallels  the  process  model  described  above.  That  is  we  ask 
ourselves  what  tasks  does  the  decision  maker  engage  in  when  performing 
the  functions  show  in  our  first  model.  Figure  3.2  shows  how  the  tasks 
map  onto  the  process  stages.  Thus  during  the  problem  recognition  and 
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Figure  3.2.  Mapping  from  Decision  Stages  to  Decision  Tasks 
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problem  structuring  phases,  the  decision  maker  is  primarily  engaged  on 
tasks  of  information  gathering  and  model  building.  At  the  point  at 
which  he  generates  the  options,  ve  can  imagine  he  i6  engaged  on  a  task 
of  creative  problem  solving.  Choice  of  option  is  supported  by  careful 
assessment  of  outcome  values  and  probabilities.  Finally,  implementation 
and  review  requires  the  decisionmaker  both  to  prepare  and  issue  orders 
and  to  get  specific  information  about  the  evolution  of  his  plan. 

This  task  model  gives  us  a  structure  for  understanding  why  the 
overall  decision  process,  and  option  generation  in  particular,  may  not 
be  performed  effectively.  First,  the  decision  maker  may  simply  not  be 
getting  enough  relevant  information  about  the  problem.  He  may  not  even 
be  aware  that  a  problem  has  arisen  until  it  is  too  late  for  him  to 
generate  an  appropriate  response.  Second,  he  may  not  be  making  adequate 
use  of  the  information  he  does  have.  Thus,  for  example,  the  model  he 
constructs  of  the  problem  may  not  include  important  elements.  This 
could  be  through  ignorance,  but  more  likely  through  oversight.  Even  if 
his  view  of  the  problem  is  adequate,  the  decision  maker  may  not  be  able 
to  define  a  solution.  The  deficiency  here  is  one  of  under-utilization 
of  creativity  skills.  It  is  during  decision  analysis  that  the  majority 
of  cognitive  biases  operate.  These  act  primarily  to  reduce  the 
efficiency  with  which  information  is  gathered  and  processed.  Difficulty 
in  choosing  an  option  for  implementation  evan  after  a  formal  decision 
analysis  may  be  due  lack  of  confidence  in  the  models  used  for  the 
analysis.  Penultimately,  the  decision  may  be  ineffectively  implemented 
because  of  inappropriately  formulated,  or  poorly  communicated,  plans. 
Then  finally,  monitoring  may  also  be  defective,  leading  to  missed  cues 
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and  inadequate  responses  to  changes  in  the  problem  context. 


Most  of  these  potential  reasons  for  poor  decision  making  stem  from 
the  human  decision  maker's  cognitive  limitations.  It  must  be, 
therefore,  that  if  we  are  to  help  then  we  should  provide  tools  which  can 
be  used  to  offset  this  weakness.  Since,  as  we  have  mentioned,  there  are 
no  predictive  theories  of  cognitive  weakness  we  must  rely  on  providing  a 
set  of  tools  from  which  the  decision  maker  can  select  and  use  as 
necessary.  We  will  describe  the  contents  of  the  toolbox  in  a  later 
section,  and  for  now  ask  what  we  must  do  to  encourage  the  decision  maker 
to  recognize  his  limitations  and  work  with  the  tools  we  can  provide. 
That  is,  we  need  to  consider  the  environment  in  which  the  decision  maker 
is  to  operate. 

3.3.  Creativity  and  Option  Generation 

In  our  discussion  so  far,  we  have  shown  that  options  generation  is 
central  to  the  decision  making  process.  We  have  also  shown  that  it  is 
not  possible  to  consider  the  generation  of  option  in  isolation,  and 
have  considered  some  reasons  why  performance  on  this  task  may  be 
inadequate.  Underlying  this  discussion  is  the  basic  belief  that,  since 
the  decision  maker  is  the  ultimate  source  of  all  option  ,  the  function 
of  our  proposed  environment  is  to  enhance  his  ability  to  generate  a  rich 
set  of  options.  If  we  take  this  point  of  view,  then  we  need  to  examine 
carefully  the  nature  of  the  environment  we  create.  We  need  to 
understand  both  its  general  characteristics  and  its  specific  components, 
and  we  need  some  psychological  theory  to  help  us  build  a  secure 
foundation  for  our  design. 


21 


V  — 


In  the  kind  of  open  decision  problem  in  which  we  are  interested, 
the  decision  maker  is  usually  faced  with  a  set  of  complex  and  unusual 
circumstances .  His  task  is  to  find  potential  solutions  despite  this 
complexity,  and  it  will  be  characteristic  of  these  solutions  that  they 
both  draw  together  diverse  aspects  of  the  problem  and  are  insightful. 
Another  way  of  describing  such  behavior  would  be  to  label  it  "creative", 
and  we  believe  that  by  doing  so  we  capture  the  essence  of  the  option 
generation  task. 

"The  generation  of  alternatives  ...  require.(s)  much  imagination 
and  creative  thinking." 

Hogarth  and  Makridakis[32] 

The  concept  of  creativity  as  a  cognitive  process  has  been  studied 
extensively,  and  there  have  been  many  attempts  to  define  what  it  is  to 
be  creative  and  how,  if  at  all,  creative  behavior  can  be  encouraged. 
Unfortunately,  there  does  not  appear  to  be  a  consensus  among  those 
working  in  the  area.  This  situation  is  exacerbated  by  the  corresponding 
lack  of  any  convincing  experiments;  most  claims  being  based  upon 
anecdotal  evidence  and  subjective  testing.  However,  an  attempt  to 
integrate  and  explain  all  the  apparently  conflicting  views  of  creativity 
has  recently  appeared,  Amabile[5],  and  in  this  model  we  find  what  we 
need  to  help  us  define  our  option  prompting  environment. 

The  key  to  Amabile's  model  lies  in  its  recognition  that  both 
motivation  and  problem  domain  knowledge  are  necessary  if  the  decision 
maker  is  to  make  effective  use  of  his  inherent  creativity  skills.  Thus 
we  have  a  "ccmponential  model  of  creativity"  as  shown  in  Figure  3.3 
where  we  indicate  how  the  three  major  components  interact  with  one 
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Figure  3.3.  A  Componential  Model  of  Problem  Solving 


another  and  with  the  problem.  The  fine  structure  of  this  model  is  not 
yet  worked  out,  but  in  our  search  for  some  insight  into  the  nature  of 
the  options  prompting  environnent  it  is  not  necessary  that  we  have  a 
detailed  psychological  description  of  how  skills  are  learned  or  how 
they  interact  with  motivation.  All  we  need,  in  fact,  is  a  coarse 
understanding  of  the  main  components  and  how  they  are  interconnected. 

Task  motivation  has  powerful  effects  on  performance,  although  the 
exact  mechanism  is  unknown.  It  accounts  for  the  motivational  variables 
that  determine  the  decision  maker's  approach  to  a  given  task,  such  as: 
initial  level  of  intrinsic  motivation,  presence  or  absence  of  extrinsic 
social  constraints,  and  individual  ability  to  minimize  the  cognitive 
effects  of  extrinsic  constraints.  One  interpretation  of  the  effects  of 
motivation  is  that  it  changes  people's  level  of  physiological  arousal. 
For  example,  a  sleepy  person  will  actually  become  more  aroused  if  he 
begins  to  perform  a  task.  The  Yerkes-Dodson  Law  [86]  states  that 
performance  is  a  curvilinear  function  of  arousal;  that  is,  performance 
initially  improves  with  increasing  arousal,  but  will  deteriorate  if 
arousal  continues  to  increase.  Therefore,  if  people  are  overaroused, 
perhaps  by  excessive  motivation,  the  likelihood  that  they  will  think  of 
a  variety  of  solutions  to  a  problem  decreases.  Potential  problem 
solving  aids  could  focus  on  increasing  or  decreasing  motivation,  to 
maintain  it  an  an  optimum  level. 

Domain-relevant  knowledge  is  a  limiting  factor  in  the  types  of 
option  people  can  generate.  It  includes  factual  knowledge  about  the 
problem,  technical  skills  and  special  talents.  Clearly,  people  cannot 
arrive  at  a  solution  which  depends  upon  knowledge  they  do  not  possess. 
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although  they  may  generate  a  partial  solution,  recognize  the  gap  in 
their  knowledge  base,  and  seek  out  the  necessary  missing  information. 
However,  domain  relevant  knowledge  is  not  limited  to  sets  of  facts. 
Experts  do  not  simply  know  more  than  novices.  Experts  and  novices  begin 
their  problem  representations  with  different  problem  categories,  and  the 
complete  the  problem  by  retrieving  knowledge  associated  with  the 
categories.  For  example,  experts  in  physics  categorize  physics  problems 
as  examples  of  abstract  physics  principles,  whereas  novices  ba6e  their 
representation  of  the  problem  and  subsequent  problem  solving  approach  on 
the  problem's  literal  features  (Chi,  Feltovich,  and  Glaser  [12]).  The 
expert's  qualitative  knowledge  is  typically  tacit,  and  cannot  be  quickly 
transmitted  to  novices. 

The  third  component,  creativity  skills,  includes  cognitive  style, 
implicit  or  explicit  knowledge  of  heuristics  for  generating  novel  ideas 
and  working  style.  The  decision  maker  may  have  a  collection  of  general 
tactics  which  he  applies  to  problems,  such  as  thinking  backward  from  the 
goals  to  the  given  state,  breaking  up  the  problem  into  subproblems, 
generating  alternative  representations  of  the  problem,  or  searching  for 
an  analogous  problem  which  has  already  been  solved  (Wickelgren  [80]). 
Creativity  skills  appear  to  be  the  component  most  amenable  to  external 
aids.  Although  nmnerous  books  are  available  on  how  to  stimulate 
creativity  and  problem  solving  (Adams  [2],  Osborne  [57],  Gordon  [2b]), 
their  impact  has  not  been  great.  First,  most  of  the  methods  ot 
increasing  creativity  or  creative  problem  solving  have  not  been 
evaluated  experimentally  (Campbell  [11]).  Studies  that  have  been  done 
have  mainly  studied  artificial  problems,  such  a6  uses  for  a  wire  coat 
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hanger.  An  exception  is  a  study  by  Basadur,  Graen  and  Green  [9]  which 


found  improved  problem  finding  by  engineers  given  a  two  day  creativity 


training  seminar,  but  they  did  not  assess  actual  problem  solving 


performance.  One  problem  of  extensive  creativity  training  programs  is 


that  the  results  may  not  generalize  to  performance  outside  the  training 


program.  When  the  facilitation  and  guidance  of  the  program  leader  is  no 


longer  available,  people  may  be  unable  or  unwilling  to  use  the 


suggestions  when  they  are  actually  confronted  with  a  problem. 


This  description  is  not  meant  to  be  a  complete  exposition,  but  is 


merely  meant  to  illustrate  Amabile's  claim  that  the  three  components  do 


form  a  necessary  and  sufficient  set  of  factors  for  understanding 


creative  behavior.  Thus  in  terms  of  Figure  3.3,  we  see  that  without  a 


sufficient  level  of  motivation  the  decision  maker  will  not  even  begin  to 


consider  the  problem.  Once  he  does  though,  he  needs  to  have  both  domain 


skills  and  creativity  skills.  To  some  extent  these  can  both  be  learned, 


but  both  will  be  applied  more  effectively  if  task  motivation  i6  high. 


All  the  components  interact  with  the  problem,  and  as  the  decision  maker 


perceives  himself  moving  towards,  or  away  from,  his  goal,  or  sub-goal, 


then  his  level  of  motivation  will  be  modified  thus  providing  the 


necessary  feedback  to  drive  the  problem  solving  process. 


Even  at  this  macroscopic  level  we  can  draw  some  inferences  from  the 


model  about  the  properties  our  environnent  should  possess.  First,  it 


must  help  the  decision  maker  maintain  his  interest  in  the  problem.  It 


might  do  this  by  ensuring  that  there  are  several  tools  available  for  him 


to  use  for  any  given  sub-task  that  he  is  engaged  upon.  Second,  the 


environment  should  allow  him  to  make  the  best  use  of,  and  maybe  even 
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improve,  both  his  domain  and  creativity  skills. 


We  can  get  even  more  insight  by  considering  hov  the  components  of 
Amabile's  model  interact  with  the  stages  in  our  process  model  of 
decision  making.  This  interaction  is  shown  in  Figure  3.4.  We  have 
collapsed  the  implementation  and  review  stages  into  one  called  "Outcome" 
to  emphasize  that  the  feedback  to  motivation  occurs  at  several  points. 
Notice  that  both  creativity  skills  and  task  motivation  directly  affect 
option  generation.  This  implies  that  if  the  environment  is  to  help  at 
all  at  this  crucial  stage,  it  can  do  this  best  by  making  the  decision 
maker  engage  in  activities  that  help  him  "break  set".  That  is  tasks 

designed  to  aid  him  view  the  problem  from  many,  maybe  superficially 
contradictory  or  ridiculous,  perspectives.  This  mapping  from  the 
psychological  framework  to  the  decision  process  framework  also  indicates 
that  the  specific  tools  we  provide  may  be  categorized  not  only  by  the 
task  that  they  help  support,  but  also  by  those  components  of  Amabile's 
model  that  they  most  address.  The  possibility  of  providing  tools  that 
only  help  develop  skills,  or  increase  motivation,  thus  arises.  We 
discuss  these  ideas  in  more  detail  in  the  next  section. 

To  conclude  this  section  we  consider  the  problem  of  measuring 
creativity.  Most  writers  on  the  subject  of  creativity  define  it  in 
terms  of  the  end  product.  That  is,  given  a  problem,  a  solution  is 

creative  if  it  is  sufficiently  different  from  what  has  been  suggested  in 
the  past  and  is  at  the  same  time  workable.  This  definition  is 
satisfactory  in  so  far  as  it  applies  to  problems  that  have  clear 

solutions.  It  is  typical,  for  example,  of  the  tests  a  patents  officer 

might  apply  to  a  new  device  that  is  given  to  him  for  possible  licensing. 
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Unfortunately,  the  concepts  of  novelty  and  feasibility,  which  are  vague 
even  for  clearly  defined  problems,  became  increasingly  imprecise  when  we 
attempt  to  apply  them  to  our  typical  open  problem.  Furthermore,  the 
nature  of  the  option  generation  task  is  such  that  we  are  also 
interested  in  the  set  of  options.  Thus  not  only  do  we  concern  ourselves 
with  the  characteristics  of  the  individual  options,  but  we  are  also 
concerned  with  properties  of  the  ensemble  of  options.  Typically  we 
might  be  interested  in  the  number  of  options  generated  and  their  breadth 
of  coverage  of  the  major  issues  in  the  problem.  The  main  point  to  make 
here  is  that  the  measurement  problem  is  rather  severe.  It  is  naive  to 
imagine  that  a  complex  concept  can  be  captured  with  a  single  numerical 
measure.  At  the  very  least  we  need  a  vector  of  measures,  some 
components  of  which  can  only  be  subjectively  determined.  We  develop 
this  view  in  Chapter  6  where  we  define  and  use  a  measuring  scheme  for 
option  generated  by  the  subjects  in  our  second  experiment. 

3.. 4.  Cognitive  Biases  and  Styles 

In  defining  our  environment  we  have,  so  far,  considered  an  abstract 
model  of  the  decision  process,  a  corresponding  task  model  and  a 
psychological  model  of  creativity.  We  now  focus  on  cognitive  biases  and 
styles  as  representing  one  further  set  of  psychological  factors  that 
should  be  taken  into  account.  We  ask  what  impact  biases  and  styles  can 
have,  and  examine  ways  in  which  the  environment  can  be  designed  to 
minimize  bias  effects  and  enhance  the  strengths  contained  in  various 
styles. 

While  cognitive  biases  have  received  much  attention  from 
experimental  psychologists,  our  understanding  of  than  is  rather  limited. 
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See  Kahneman  et  al  [45]  for  recent  views  of  these  issues.  So  that 
although  we  can  enumerate  a  large  variety  of  systematic  biases  that  can 
occur,  we  have  no  theory  which  tells  us  if  indeed  they  will  occur  under 
any  given  conditions.  This  makes  it  difficult  for  us  to  design  an 
environment  that  will  successfully  prevent  the  decision  maker  from 
exhibiting  them.  The  best  we  can  do  is  to  be  alert  for  typical 
situations  in  which  biases  are  known  to  occur  and  then  make  the  decision 
maker  aware  that  biases  may  be  operating.  Specific  techniques  for  bias 
correction  have  not  been  reported,  except  for  some  special  cases  noted 
by  Kahneman  and  Tversky[44],  so  the  help  the  environment  gives  must  be 
of  a  rather  general  form. 

One  way  of  exploring  the  potential  impact  is  to  categorize  biases 
according  to  the  decision  process  stages  in  which  they  occur.  We  show, 
in  Table  3.1,  a  list  of  commonly  known  biases  organized  by  process 
phase.  We  see  that  biases  that  affect  the  way  in  which  the  decision 
maker  reacts  to  cues  in  the  environment  have  most  impact  on  the 
information  gathering  task.  Biases  that  affect  the  way  in  which 
information  is  processed  have  most  impact  on  that  part  of  model  building 
and  problem  solving  which  requires  inference,  reasoning  and  estimation. 
Biases  that  affect  the  way  in  which  judgements  are  expressed  have  most 
impact  on  outcome  evaluations,  and  biases  that  affect  the  way  in  which 
the  decision  maker  gets  feedback  from  the  problem  environment  have  most 
impact  on  his  ability  to  implement  and  review  the  chosen  plan. 

By  doing  the  categorization  this  way,  we  can  get  some  ideas  about 
general  environnent  characteristics.  Thus  everywhere  ve  know  that  a  bias 
might  exist,  we  should  try  to  ensure  that  the  environment  minimizes  the 
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Table  3.1.  Some  Common  Biases 


likelihood  that  it  will  actually  occur.  If  the  decision  maker  is 
searching  for  information  then  we  should  try  to  make  sure  he  sees 
everything  relevant,  not  just  that  which  he  thinks  is  relevant.  So,  for 
example,  if  the  decision  maker  is  assessing  probabilities  then  we  should 
ensure  that  he  is  using  base  rate  information  correctly.  The  question  of 
how  we  might  make  such  procedures  operational  is,  of  course,  still  open, 
but  we  will  discuss  a  possible  approach  in  the  final  section  of  this 
chapter. 

In  contrast  to  the  widely  accepted  phenomenon  of  a  cognitive  bias, 
the  existence  of  cognitive  (or  decision)  style  is  highly  contentious. 
Although  the  concept  of  a  decision  maker  having  characteristic  patterns 
of  behavior  within  the  problem  solving  context  is  uncontroversial ,  the 
idea  that  these  patterns  are  rigid  and  that  decision  makers  fall  into 
well  defined  categories  has  met  with  considerable  resistance.  See 
Hubert 36]  for  a  good  discussion  of  some  of  these  points,  and  Zmud[87] 
for  a  review  of  the  experimental  data.  More  importantly,  there  is  no 
real  evidence  that  possession  of  a  specific  style  in  any  way  helps  or 
hinders  the  process  of  decision  making.  Nor  does  there  appear  to  be  any 
work  that  shows,  or  even  attempts  to  show,  correlations  between  style 
and  biases. 

If  cognitive  style  has  any  impact  at  all  on  our  concept  of  an 
environment,  then  we  believe  it  to  be  of  only  secondary  importance.  We 
need  to  be  aware  of  typical  behaviors  so  that  we  provide  as  much 
flexibility  as  possible,  both  in  terms  of  the  features  in  the 
environment  and  the  way  in  which  the  decision  maker  can  interact  with 
them,  but  we  view  this  as  an  implementation  rather  than  a  design  issue. 
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Table  3.2.  Examples  of  Tools 


While  our  argument  is  that  the  decision  maker  may  require  help  in 
performing  all  these  tasks,  in  our  current  effort  we  have  concentrated 
on  those  in  which  generic,  rather  than  problem  specific,  tools  are  most 
appropriate.  We  have,  therefore,  been  mainly  concerned  with  model 
building  and  problem  solving.  The  details  of  our  efforts  in  these  areas 
are  given  in  the  following  three  chapters.  The  remainder  of  this  section 
is  concerned  with  giving  a  survey  of  the  tool  categories  described 
above. 

Information  gathering  is,  obviously,  the  task  of  obtaining 
knowledge  about  the  problem  and  its  context.  The  decision  maker  might 
make  use  of  a  wide  range  of  sources:  from  computerized  data  bases  to 
card  indexes,  from  reference  books  to  telephone  conversations  with 
experts.  All  these  sources  may  not  be  available  in  the  decisionmaker's 
immediate  physical  location,  so  our  concept  of  environment  here  is 
somewhat  vague.  All  that  is  important  is  that  the  decision  maker  has 
ready  access  to  the  information  he  feels  is  relevant.  Clearly,  the  kind 
of  information  required  is  problem  dependent  and  it  has  not  seemed 
feasible  in  this  study  to  examine  these  information  requirements  in 
great  detail.  However,  we  note  that  in  the  business  world  the  supply  of 
information  to  management  through  computerized  management  information 
systems  is  seen  as  an  important  aid  in  improving  the  quality  of  decision 
making. 

Model  building  also  covers  a  wide  range  of  activities.  It  is, 
however,  more  amenable  to  general  approaches  than  is  information 
gathering.  Thus  there  are  some  general  techniques  that  can  be  applied 
without  direct  reference  to  the  particular  problem  faced  by  the  decision 
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maker.  In  our  work  on  option  generation,  it  has  been  this  issue  of 
model  building  that  has  most  concerned  us.  We  will  elaborate  on  the 
reasons  for  this  in  the  next  chapter,  but  it  seemed  to  us,  at  least 
initially,  that  in  complex  problems  what  the  decision  maker  lacks  most 
is  the  facility  for  structuring  the  main  elements  of  the  problem. 

Tools  that  might  be  appropriate  in  helping  the  decision  maker 
construct  a  representation  of  the  problem  range  from  simple  check  lists 
to  more  formal,  and  elaborate,  schemes  such  as  AHP.  Of  course,  each 
structuring  tool  has  its  appropriate  use;  a  checklist  might  be  useful  if 
only  a  limited  amount  of  time  is  available,  or  if  the  decision  maker 
only  needs  a  preliminary  structure  for  his  problem.  More  elaborate 
tools  require  the  decision  maker  to  expend  greater  effort  and  this  may 
not  always  be  appropriate  or  convenient. 

Tools  for  problem  solving  are  hard  to  define  since  we  have  no 
detailed  model  of  this  cognitive  process.  Nonetheless,  we  do  have  a 
qualitative  understanding  and  this  can  be  used  to  help  us  construct  a 
variety  of  potentially  useful  tools.  In  doing  so  it  is  important  to 
realize  that  there  are  several  methodologies  in  existence  for  improving 
creativity.  Some  of  the  best  best  known  of  these  are  Conceptual 
Blockbusting[2] ,  Lateral  Thinking] 10] ,  Brainstorming] 57]  and  Synectics 
[63],  and  within  them  there  are  some  specific  procedures  which  are 
claimed  to  improve  problem  solving  ability.  The  main  characteristic  of 
all  these  techniques  is  that  they  attempt  to  make  the  problem  solver 
take  an  unusual  view  of  the  problem.  They  use  assumption  challenging, 
unrestricted  non- judgemental  thinking  and  reasoning  by  analogy  as  ways 
of  aiding  the  breaking  of  mind  sets.  We  believe  that  such  devices  can 
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be  easily  incorporated  in  the  toolbox. 

Tools  for  helping  the  decision  maker  perform  evaluations  of  options 
exist  in  many  forms.  Perhaps  the  most  widely  known  are  those  in 
decision  analysis.  However,  in  our  view  of  the  problem  any  device  that 
provides  information  about  the  likely  effect  of  implementing  the  options 
is  an  outcome  evaluator.  Thus  a  computer  simulation  model  would  be 
included  in  this  category,  as  well  as  AHP,  decision  trees  and  multi¬ 
attribute  utility  methods. 

Finally,  monitoring  tools  would  include  devices  for  recording  the 
progress  of  a  plan,  techniques  for  detecting  deviations  from  expected 
paths  and  methods  for  deciding  how  plans  should  be  changed  to  take 
account  of  new  information.  In  our  effort,  we  have  not  addressed  these 
issues  at  all.  The  reason  being  that  we  have  focussed  on  the  early 
stages  of  the  decision  making  process,  believing  that  most  insight  and 
progress  would  be  achieved  by  considering  the  relatively  unexplored 
issues  of  structuring  and  creativity.  Also,  since  we  have  been 
constrained  to  work  on  generic  open  problems,  rather  than  specific 
problem  domains,  it  is  natural  that  we  should  have  avoided  exploring 
tools  that  need  this  special  domain  knowledge. 

3. 6.  A  System  Architecture  for  Option  Promoting 

In  this  final  sub-section  we  describe  a  possible  computer  system 
architecture  that  would  support  our  concept  of  an  options  prompting 
enviroment.  This  serves  two  purposes.  First,  it  enables  us  to  draw 
together  the  ideas  discussed  in  the  earlier  sections,  and  second,  it 
shows  how  our  environment  could  be  made  operational.  The  intention  is  to 
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provide  just  the  outline  of  a  system  and  to  indicate  hov  it  would  be 
used. 

A  block  diagram  of  the  proposed  system  is  shown  in  Figure  3.5.  It 
contains  five  major  elements:  a  user  interface,  a  knowledge  base,  an 
executive,  an  option  list,  and  a  toolbox.  The  last  of  these  is  the 
collection  of  tools  we  discussed  in  the  previous  section.  Fran  the 
system  point  of  view,  the  toolbox  will  also  contain  simple  tutorial 
material  about  the  tools  and  their  advantages  and  disadvantages.  This 
allows  the  decision  maker  to  explore  the  capabilities  of  the  toolbox  if 
he  is  unsure  of  what  it  contains  or  needs  help  in  using  a  specific  tool. 

The  toolbox  by  itself  is  not  sufficient  for  us  to  provide  the 
guidance  and  help  the  decision  maker  actually  needs  in  choosing 
appropriate  tools  and  overcoming  potential  biases.  To  give  the 
environment  the  desired  characteristics,  we  have  to  provide  a  repository 
for  our  understanding  of  typical  decision  behavior  (the  knowledge  base), 
a  mechanism  for  utilizing  this  knowledge  (the  executive),  a  device  for 
recording  the  option  structure  being  developed  (the  option  list),  and 
procedures  that  allow  the  decision  maker  to  interact  easily  with  the 
system  (the  user  interface). 

If  we  think  of  the  environment  that  we  construct  as  having  a 
persona,  then  our  aim  is  to  simulate  an  "objective  critic".  The 
environment  provides  help  when  prompted  by  the  user,  but  also  has  the 
capacity  to  interject  when  biases,  or  potential  biases,  or  inappropriate 
behavior  are  detected.  Thus  the  role  of  the  enviroanent  is  to  remain 
passive  so  long  as  the  decision  maker  is  satisfied  with  his  progress. 
When  he  needs  help,  the  environment  should  be  able  to  give  him  advice 
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and  point  out  possible  flaws  in  his  current  solution  set. 

The  key  element  in  our  architecture  is  the  knowledge  base.  Without 
it  our  environment  would  be  entirely  passive  and  not  significantly 
different  from  a  simple  toolbox.  It  is  within  the  knowledge  base  that  we 
keep  our  high  level  knowledge  about  decision  making.  This  can  take  many 
forms;  from  simple  prescriptions  for  giving  responses  to  "what  next" 
questions  from  the  decision  maker,  to  elaborate  procedures  for  checking 
whether  the  decision  maker  is  engaging  in  unsatisfactory  behavior. 

The  purpose  of  the  executive  i6  to  perform  all  interpretive  and 
housekeeping  tasks  that  are  necessary  to  make  the  system  work.  It  keeps 
track  of  where  the  decision  maker  is  in  relation  to  his  use  of  the 
tools,  it  decodes  and  acts  upon  commands  given  by  the  user  and  it 
generally  controls  the  flow  of  data  in  the  system. 

The  idea  behind  the  current  option  list  is  that  the  decision  maker 
needs  some  mechanism  for  storing  his  currently  active  set  of  options. 
This  may  be  a  simple  list  but,  more  likely,  it  will  have  some  form  of 
tree  structure.  The  enviroixnent  will  allow  the  decision  maker  to  record 
his  options  in  the  most  appropriate  form  and  provide  a  simple  way  of 
editing  them.  In  its  more  active  mode,  the  enviroonent  may  have  the 
ability  to  detect  that  the  options  being  generated  are  merely 
refinements  of  existing  options.  It  could  then  suggest  to  the  decision 
maker  that  efforts  be  made  to  think  of  options  in  other  areas. 
Conversely,  if  the  decision  maker  has  thought  of  several  high-level 
options  but  has  not  begun  to  explore  the  details  of  any  of  them,  then 
the  environment  could  remind  him  that  this  might  be  a  valuable  way  to 
proceed.  The  option  list  serves  two  purposes,  therefore.  It  allows  the 

40 


decision  maker  Co  record  bis  ideas,  but  it  also  provides  a  way  of 
Celling  Che  environment  how  the  decision  maker  is  progressing.  As  the 
decision  proceeds,  then  the  option  list  will  become  smaller  until,  at 
the  point  of  decision,  it  is  reduced  to  a  single  entry. 

The  user  interface  is,  as  its  name  implies,  that  part  of  the 
environment  that  the  decision  maker  interacts  with.  Its  purpose  is  to 
make  the  process  of  using  the  environment  as  easy  and  acceptable  as 
possible.  The  interface  will  allow  the  decision  maker  to  have  a  flexible 
dialog  with  the  system  by  utilizing  advanced  graphics  and  interactive 
terminals. 

Overall  our  architecture  is  designed  to  provide  a  decision 
environment  that  is  supportive,  helpful  and  conducive  to  generating  good 
solution  options  to  open  decision  problems.  It  exists  only  on  paper  of 
course,  but  we  have  tried  to  show  what  the  system  requirements  are  and 
how  these  reflect  our  study  of  the  issues.  In  the  next  three  chapters  of 
the  report  we  will  describe  our  attempts  to  explore  some  of  the  details 
of  our  environment.  We  discuss  a  particular  form  of  structuring  tool  and 
an  experiment  designed  to  test  its  value,  and  then  describe  an 
experiment  that  tests  a  very  simple  form  of  environment. 
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4.  THE  ANALYTIC  HIERARCHY  PROCESS  FOR  DECISION  PROBLEM 


STRUCTURING  AND  DECISION  FACTOR  PRIORITIES 


In  the  previous  chapter  ve  described  the  concept  of  an  Option 
Prompting  Environment.  One  of  the  underlying  motivations  for  this  view 
of  the  options  generation  problem  was  the  belief  that  decision  making  is 
a  multistage  iterative  process.  Problem  structuring  and  determining  the 
importance  of  decision  factors  were  then  identified  as  two  important 
activities  that  the  decision  maker  could  perform  to  focus  on  those  areas 
and  relationships  most  likely  to  lead  to  new  germane  alternatives.  In 
this  chapter  we  examine  the  issues  of  problem  representation  and 
structure,  and  present  Saaty' 8  "Analytic  Hierarchy  Process"  [67]  as  a 
powerful  aid  for  helping  the  decision  maker  order  and  analyze  the 
factors  that  will  influence  his  decision.  Appendix  C  contains  a 
detailed,  mathematical  description  of  the  AHP. 


Problem  Representation 


Decision  problems  often  require  the  decision  maker  to  identify  and 
organize  complex  sets  of  goals,  actors,  constraints,  and  resources.  If 
some  of  these  are  inadvertently  ignored,  or  the  relationships  between 
them  not  properly  understood,  then  potential  solutions  to  the  problem 
may  be  overlooked.  In  conventional  decision  analysis  this  stage  of  the 
decision  process  is  called  problem  structuring,  or  issue  identification. 
Its  importance  is  recognized  by  most  decision  analysts,  with  the  result 
that  several  major  software  packages  have  been  developed  to  help  the 
decisionmaker  with  this  task  (e.g.,  Gulick  [27],  Humphreys  et  al  [39], 
Merkhoffer  et  al  [33]).  However,  in  all  of  these  studies  there  is  an 
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implicit  assumption  that  the  problem  can  be  represented  by  means  of  the 
classical  decision  tree.  This  approach  presupposes  that  models  with 
sufficient  fidelity  to  the  specific  decision  can  be  constructed  for  each 
branch  of  the  tree  and  used  to  calculate  values,  which  in  turn  can  be 
used  in  a  procedural  way  to  select  the  preferred  option.  However,  in 
complex,  novel,  open  decision  situations,  it  appears  that  this  is  an 
invalid  assumption,  or  at  least  a  considerable  amount  of  effort  is 
required  before  appropriate  models  and  values  can  be  constructed.  Time 
may  not  be  available. 

There  is  evidence  (see  [60], [61])  to  suggest  that,  at  least  in  the 
preliminary  stages  of  problem  structuring,  decision  makers  would  achieve 
a  better  understanding  by  concentrating  on  their  own  goals  and  other 
major  determinants  in  the  problem,  rather  than  by  trying  to  develop  a 
formal  time  evolving  sequence  of  action/outcome  alternatives.  Indeed, 
Pearl  et  al  [59]  use  exactly  this  argument  to  justify  their  development 
of  a  structuring  aid  that  helps  the  decision  maker  develop  a  tree  of 
goals  and  sub-goals. 

Saaty  [67]  notes  that  significant  factors  may  be  left  out  because 
of  simplifying  assumptions  needed  to  make  the  quantitative  models 
tractable.  He  states,  "To  be  realistic,  our  models  must  include  and 
measure  all  important  tangible  and  intangible,  quantitatively 
measurable,  and  qualitative  factors."  The  AHP  provides  the  mechanisms 
for  doing  this.  Saaty  provided  a  program  for  the  mathematics  of  the 
analytic  assessment  of  priorities.  AI&DS  has  extended  this  program  to 
also  help  facilitate  construction  of  an  appropriate  hierarchy  model  for 
the  decision  problem. 
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Within  the  context  of  the  Option  Prompting  Environment  discussed 
in  the  last  chapter,  the  toolbox  should  contain  a  selection  of  tools  for 
problem  structuring  and  evaluation.  The  specific  tool(s)  selected  by 
the  decision  maker  will  be  specific  decision  problem  dependent.  We 
believe  that  focusing  on  goals,  actors,  constraints,  and  resources  is  a 
very  general  paradigm  that  can  be  applied  to  many  decision  situations. 
We  further  believe  that  AHP  is  a  very  flexible  and  powerful  tool  for 
making  this  paradigm  operational. 


4.2.  The  Analytic  Hierarchy  Process 


There  are  two  major  activities  involved  in  using  the  AHP:  problem 
structuring  and  decision  factor  prioritization.  Each  of  these 
subprocesses  can  be  accomplished  to  varying  degrees  of  detail  and 
accuracy,  depending  on  the  decisionmaker's  desires,  time,  and  analysis 
resources  available.  The  problem  structuring  results  in  a  multilevel 
hierarchy  of  the  decision  factors,  each  of  which  can  be  quantitative  or 
qualitative  in  nature.  Prioritization  of  the  decision  factors  results 
in  an  assessment  of  the  importance  of  each  of  the  decision  factors,  both 
with  respect  to  the  overall  goal  of  the  decision,  and  with  respect  to 
other  related  decision  factors  in  an  adjacent  level  of  the  hierarchy 
(i.e.,  the  other  decision  factors  that  a  given  decision  factor  is  most 
likely  to  directly  affect). 


4.2.1.  Decision  problem  structur ins 


When  confronted  with  a  complex  situation  that  requires 
understanding  and  a  decision,  a  decision  maker  is  likely  first  to  gain 
an  intuitive  understanding  of  the  situation,  and  then  perhaps  proceed 
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toward  a  more  quantitative  model  of  appropriate  parts  of  the  situation. 
A  mechanism  for  gaining  an  understanding  of  the  decision  situation  is 
through  the  construction  of  a  hierarchy  containing  all  the  elements,  or 
decision  factors,  arranged  in  appropriate  groupings,  i.e.,  levels  of  the 
hierarchy. 

There  are  several  mechanisms  for  constructing  a  hierarchy  that 
represents  a  "soft"  model  of  the  decision  situation.  Saaty  [67] 
believes  there  is  an  innate  method  of  the  human  mind  that  groups  a  list 
of  elements  which  comprise  a  complex  situation  into  groups,  according  to 
whether  they  share  certain  properties.  The  common  properties 
identifying  one  level  of  the  hierarchy  (group  of  elements)  can  be  taken 
as  the  elements  of  a  new  level  in  the  system.  This  grouping  process  can 
be  recursively  applied  until  we  reach  either  the  single  "top"  element 
(which  is  often  the  overall  goal  of  the  decision  process)  or  the 
"bottom"  level  of  basic  ingredients  of  the  decision  process  (which  are 
often  identified  as  the  resources  that  can  be  controlled).  This  process 
results  in  the  elements  in  adjacent  levels  being  related  through  the 
defining  properties  for  the  levels.  This  set  of  relationships 
represents  and  the  constructed  hierarchy  provide  structure  for  the 
problem.  In  practice,  this  structure  serves  as  a  vehicle  for  analyzing 
the  functions  of  the  systems  within  the  decision  situation. 

Saaty  [67]  has  applied  this  hierarchical  structuring  (modeling) 
process  in  several  application  domains  and  presents  generic  formats  for 
the  different  types  of  domains.  For  high  level  modeling  of  (military) 
conflict  situations  we  believe  the  generic  hierarchy  template  of  Figure 
4.1  is  appropriate.  (The  identifying  descriptors  for  each  level  of  the 
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Figure  4.1  Generic  Hierarchy  (Template) 
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hierarchy  imply  the  types  of  elements  that  would  be  put  into  that 
level. ) 


During  the  contract  the  intent  has  been  to  stay  as  generic  as 
possible.  Hence  the  above  template  is  quite  abstract.  For  a  specific 
class  of  decision  makers  we  believe  it  is  possible  to  subdivide  the 
space  of  all  types  of  decisions  that  they  are  likely  to  face,  into  a 
small  number  (on  the  order  of  10)  of  subsets.  The  decision  process 
would  be  similar  for  all  decisions  within  any  one  subset.  Thus  a 
specific  decision  situation  for  each  subset  could  be  chosen  and  a 
specific,  more  detailed  hierarchy  could  be  constructed.  The  decision 
maker  confronting  his  o_n  specific  decision  situation  could  select  the 
appropriate  template  for  his  type  of  decision,  and  using  reasoning  by 
analogy,  modify  the  template  hierarchy  to  better  fit  his  own  problem. 
Robinson  [66]  and  Kelley  et  al  146]  have  already  given  some  attention  to 
templating  for  decision  problems.  Artificial  Intelligence  techniques 
could  be  used  to  help  select  the  right  template  and  then  aid  the 
decision  maker  in  appropriately  modifying  it  for  his  specific  decision 
situation. 

For  purposes  of  option  generation  the  hierarchy  structure  required 
for  the  AHP  has  several  advantages: 

(1)  It  provides  a  mechanism  for  quickly  estimating  the  scope  of  the 
effort  that  should  be  involved  in  making  the  decison  and  a  road  map 
of  the  elements  that  deserve  at  least  some  attention.  This  may 
help  keep  the  decision  maker(s)  from  "locking  on"  one  specific 
aspect  of  the  problem  for  too  long,  to  the  detriment  of  exploring 
other  areas. 
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(2)  It  provides  a  "soft"  model  whose  structure  can  be  easily  changed  to 


explore  different  concepts. 

(3)  It  provides  a  good  vehicle  for  focusing  communication  among  various 
decision  makers  and  staff. 

(4)  It  provides  a  good  mechanism  for  ensuring  an  early  common 
understanding  of  terms  and  concepts. 

(5)  It  fosters  and  orders  discussion  of  (thinking  of)  potential 
variations  or  the  baseline  decision  situation. 

(6)  It  combines  quantitative  and  qualitative  factors  into  a  single 
structure. 

A  major  disadvantage  of  this  structure  is  that  for  some  people  it 
does  not  have  sufficient  cause  and  effect  relationships  explicitly 
defined,  i.e.,  the  model  is  not  sufficiently  well  defined  for  them. 

4.2.2.  Analytic  decision  factor  prioritization 

The  second  major  subproce6s  of  the  AHP  is  determining  the 
priorities  among  decision  factors  at  each  level  of  the  constructed 
hierarchy.  The  specific  process  is  delineated  in  detail  in  Appendix  C. 
Both  global  priorities  (with  respect  to  the  overall  goal)  and  local 
priorities  (with  respect  to  decision  factors  in  the  level  immediately 
above  the  level  of  the  hierarchy  under  current  consideration)  are 
obtained.  The  raw  input  used  to  obtain  these  priorities  is  a  sequence 
of  pair-wise  comparisons  between  decision  factors  with  respect  to 
another  factor  in  the  next  higher  level  of  the  hierarchy  —  a  factor 
related  directly  through  the  defining  properties  of  the  level  to  the  two 
decision  factors  being  compared.  Decision  maker(s)/staff  tend  to  be 

48 


willing  Co  make  these  pair-vise  comparisons  much  more  readily  than 
global  or  local  ordering  of  a  list  of  elements. 


An  important  point  for  fostering  option  generation  with  decision 
maker(s)/staff  is  the  mental  process  they  use  in  making  their  pair-vise 
comparison  estimates.  They  typically  think  of  a  myriad  of  scenario 
variations  to  arrive  at  their  estimates.  As  they  move  from  one  pair¬ 
wise  comparison  to  the  next,  they  continually  rethink  many  of  these 
scenario  situations  from  different  perspectives.  This  quick  repeated 
consideration  leads  to  identifying  why  factors  are  more  or  less 
important  and  provides  some  insight  into  what  conditions/parameters  they 
are  sensitive  to.  Thus,  they  not  only  get  an  analytic  assessment  of  the 
priorities  of  various  decision  factors,  but  gain  insight  into 
specifically  why  and  into  how  robust  or  sensitive  to  change  they  are. 

We  felt  that  understanding  the  sensitivity  to  change  and  to 
anomalies  within  the  basic  scenarios  considered  would  be  conducive  to 
generating  new  options.  During  the  contract  the  basic  AHP  theory  was 
extended  to  more  formally  make  evaluations  of  sensitivity  impacts;  these 
extensions  are  also  presented  in  Appendix  C.  However,  they  were  not 
incorporated  in  the  AHP  package  the  subjects  used  in  the  experiments 
reported  in  the  next  two  chapters. 

Major  disadvantages  of  the  prioritization  process  for  the  creative 
task  of  options  generation  is  that  it  becomes  tedious,  and  perhaps  too 
repetitive  in  nature,  for  large  hierarchies.  In  the  current  form  the 
entire  process  must  be  completed  in  a  definite  order  before  any  global 
priorities  are  produced. 


49 


A  major  payoff  for  option  generation  after  the  entire  process  is 
completed  is  that  the  most  important  decision  factors  are  indicated  and 
time  and  effort  can  be  focussed  into  those  areas  and  away  from  the  lower 


valued  factors. 


5.  THE  MONTEREY  EXPERIMENT 


In  Che  previous  chapter  we  described  Che  Analytic  Hierarchy  Process 
(AHP)  and  discussed  its  possible  use  as  a  Cool  for  helping  decision 
maker's  structure  their  thinking  about  decision  problems  and  evaluate 
the  relative  importance  of  decision  factors.  Accordingly,  our  first 
experimental  investigation  was  designed  to  evaluate  the  concept  that 
these  two  AHP  supported  processes  would  help  the  decision  maker  generate 
options.  At  the  same  time,  we  were  concerned  to  confront,  and  learn 
about,  the  difficulties  inherent  in  performing  experimental  research  in 
this  area.  A  more  detailed  report  of  the  experiment  and  its  results  can 
be  found  in  Tong[75]. 

Since  we  view  Option  Generation  as  essentially  a  creative  task  we 
are  faced  with  several  dilemmas.  We  must  first  determine  to  what  extent 
people  are  naturally  creative  when  faced  with  a  difficult  problem. 
Clearly,  understanding  normal  unaided  behavior  iB  a  pre-requisite  to  the 
development  of  devices  that  can  improve  on  this  behavior.  One  goal  of 
our  first  experiment,  therefore,  was  to  determi.ee  exactly  what  kinds  of 
options  would  be  generated. 

A  second  dilemma  is  how  to  introduce  the  aid  in  a  manner  that 
supplements  the  decisionmaker's  normal  behavior.  In  particular,  we  need 
to  guard  against  the  possibility  that  presenting  the  decision  maker  with 
an  aid  will  actually  inhibit  his  performance.  When  providing  subjects 
with  aids  that  run  as  computer  programs,  it  is  important  to  develop 
experimental  methods  that  overcome  the  subjects'  potential  difficulties 
with  the  new  technology. 
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A  third  difficulty  is  to  define  the  idea  of  a  good  option  in  a 
sufficiently  precise  manner  so  that  the  performance  of  the  subjects  in 
an  experiment  can  be  measured  and  evaluated.  Each  problem  has  many 
possible  solutions  that  may  be  presented  at  many  different  levels  of 
detail.  Determining  qualitites  such  as  the  adequacy  and  creativity  of 
options  is  a  severe  measurement  problem. 


5.1.  Experimental  Method 


A  group  of  sixteen  students  enrolled  in  the  Command  and  Control 
Curricula  at  the  U.S.  Navy  Postgraduate  School  (NPS)  in  Monterey  were 
subjects  for  our  experiment.  We  divided  them  into  two  groups  of  eight, 
attempting  to  match  the  groups  in  terms  of  service  background  and  rank. 
That  is,  we  tried  as  nearly  as  possible  to  have  an  equal  number  of  Navy, 
Air  Force  and  Army  personnel  in  each  group,  and  the  same  nunber  of 
Major/Lieutenant  Commanders  and  Captain/Lieutenants  in  each  group.  Our 
reasoning  being  that  service  training  and  background  should  have  an 
influence  on  a  subject's  ability  to  deal  with  the  problem  scenarios  we 
were  to  present.  By  partitioning  the  subjects  in  this  way  we  avoid  a 
potential  bias  in  the  results. 

The  experiment  took  place  over  two  days  at  the  NPS.  Access  to  the 

AHP  aid  was  provided  over  the  ARPANET  using  the  computer  facilities  in 
3 

the  School's  C  I  Laboratory.  The  experiment  was  an  A-B,  B'-A  design, 
with  all  subjects  being  in  a  control  condition  (A)  and  a  treatment 
condition  (B  or  B').  Thus  on  Day  1  half  the  subjects  were  in  condition 
A  and  half  in  condition  B.  On  Day  2  they  changed  round  being  in 
conditions  B'  and  A  respectively.  The  treatment  condition  on  both  days 
involved  use  of  the  AHP  aid,  but  the  subject's  interaction  with  it  was 


different  on  the  tvo  days. 


We  used  tvo  problem  scenarios:  a  "Downed  B-52"  on  Day  1,  see  Figure 
5.1,  and  a  "War  Game"  on  Day  2,  see  Figure  5.2.  These  are  primarily 
military  but  have  political  implications.  Both  groups  attended  a  brief 
verbal  introduction  to  the  problem,  were  given  the  stimulus  material  we 
prepared  (instructions  and  supplementary  material  such  as  maps)  and  were 
then  allowed  tvo  hours  to  generate  options  to  solve  the  problem. 

On  both  days,  the  subjects  in  the  control  group  worked  individually 
and  were  asked  to  write  down  as  many  creative  options  as  they  could. 
Each  option  was  to  be  written  on  a  separate  sheet  of  paper  with  no 
constraints  on  the  format. 

On  the  first  day,  the  subjects  in  the  treatment  group  worked 
individually  with  the  AHP  aid.  They  were  given  some  preliminary 
guidance  in  its  use,  but  were  then  expected  to  interact  with  the  aid  via 
a  standard  alpha-numeric  computer  terminal  to  help  produce  their  own 
written  options.  Assistance  with  the  technical  aspects  of  the  aid  was 
available  to  them  throughout  the  two  hour  period. 

On  the  second  day  our  presentation  of  the  AHP  aid  to  the  treatment 
group  was  different.  We  found  that  on  the  first  day  the  subjects  had 
difficulty  using  the  AHP  aid  because  they  were  not  familiar  with  the 
computer  protocols  needed  to  interact  with  it.  To  overcome  this,  we 
conducted  a  group  session  vith  one  of  us  acting  as  mediator  and  another 
one  of  us  as  interface  to  the  AHP  aid.  The  subjects  thus  structured  the 
problem  as  a  group,  but  then  worked  individually  to  produce  their 
written  options. 


A  fully  armed  B-52,  equipped  with  the  latest  penetration  aids,  is  on 
a  routine  peacetime  mission  at  the  Eastern  end  of  the  Aleutian  Islands. 

His  mission  profile  was  to  take  him  close  to  Soviet  territory  near  the 
Commander  Islands,  and  then  generally  Southwest,  parallel  to  the  Kuril 
Islands. 

At  1045  (local  time),  23  January,  the  B-52  reports  a  failed  generator, 
but  that  the  mission  is  being  continued.  At  1510,  a  Pan  American  jetliner, 
nominally  on  a  great  circle  route  from  Seattle  to  Tokyo,  reports  on  the 
International  traffic  control  net  that  he  has  received  a  crash  beacon  sig¬ 
nal  from  an  area  near  the  Eastern  most  Islands  in  the  Commander  Islands, 
with  a  large  associated  area  of  uncertainty. 

All  subsequent  attempts  to  contact  the  B-52  are  futile. 

There  is  ice  around  the  Commander  Islands.  Current  weather  reports 
scattered  clouds  at  3,000  feet  and  20,000  feet,  and  forecast  to  remain 
the  same  for  24  hours. 

A  Soviet  Naval  Surface  Combatant  Task  Group  is  on  maneuvers  400 
nautical  miles  to  the  Southwest. 

A  Japanese  fishing  trawler  is  250  nautical  miles  South  of  Attu  Island. 

All  U.S.  military  and  coast  guard  assets  are  conducting  their  "normal" 
peacetime  activities. 

What  should  be  done? 


Figure  5.1  Day  1  Problem  Scenario 


Grey  and  Orange  governments  have  been  ideolgically  opposed  and 
hostile  toward  each  other  for  a  long  time.  Grey's  military  capability 
has  diminished  during  the  past  few  years.  An  appreciable  segment  of  the 
population  in  Eastern  Grey  are  sympathetic  with  Orange.  Insurgency  by 
these  "Grey  Eagles,"  logistically  supported  by  Orange,  has  resulted  in 
their  de-facto  control  of  the  Eastern-most  area  of  Grey,  except  that  Grey 
still  controls  and  operates  SMALL  AIRPORT.  Recent  Grey  Eagle  successes 
encourage  Orange  to  actively  enter  the  struggle.  They  capture  SMALL  AIR¬ 
PORT  and  immediately  start  enlarging  it. 

Grey  appealed  to  the  UN  for  aid. 

Blue  has  previously  indicated  that  any  Orange  overt  use  of  force  in 
Grey  was  unacceptable,  and  asked  for  congressional  approval  for  unilateral 
support  of  Grey  if  favorable  UN  reaction  was  not  immediate.  Purple  con¬ 
curs  with  Blue. 

Blue  has  a  Tactical  Air  Wing  stationed  in  air  bases  in  Purple,  within 
range  of  SMALL  AIRPORT.  The  Wing  Commander  is  given  the  mission:  "When 
directed,  begin  operations  to  neutralize  Orange  forces  and  facilities  in 
Grey.  Do  not  attack  targets  in  Orange.  Take  defensive  measures  to  pro¬ 
tect  your  force  from  Orange  or  Red  retaliations." 

No  immediate  political  resolutions  are  made  in  the  UN  or  Blue. 

Orange  completes  a  hasty  enlargement  of  SMALL  AIRPORT.  On  1-2  May 
3  squadrons  of  Orange  MIG-21  and  3  squadrons  of  Orange  SU-7  attack  air¬ 
craft  arrive  at  SMALL  AIRPORT. 

At  2300,  2  May,  the  Wing  Commander  is  directed  to  execute  this 
assigned  mission.  The  first  reconnaissance  flight  on  3  May  reveals  the 
presence  of  several  air  defense  missle  sites  and  3  Red  Hospital  Aircraft, 
with  large  Red  Crosses,  parket  near  Orange's  aircraft  and  the  runway  at 
SMALL  AIRPORT. 

What  does  the  Wing  Commander  to  to  execute  his  mission? 


Figure  5.2  Day  2  Problem  Scenario 
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At  the  end  of  each  day's  session,  the  subjects  in  the  treatment 
group  were  asked  to  complete  a  questionnaire.  The  responses  were  used 
to  assess  the  acceptability  of  the  aid.  The  subjects  in  the  control 
group  were  asked  for  their  verbal  impressions  of  the  experimental 
procedure. 

5.. 2.  Results  and  Commentary 

Our  goals  in  this  pilot  experiment  may  be  broadly  stated  as: 

(1)  To  gain  insight  into  the  concept  of  an  option  and  how  this  depends 
on  experience  and  training, 

(2)  To  form  seme  estimates  of  the  value  of  the  AHP  aid  as  a  device  for 
helping  options  generation,  and 

(3)  To  learn  how  to  perform  an  experiment  that  attempts  to  evaluate  a 
decision  aid. 

The  data  we  collected  allows  us  to  form  seme  subjective  conclusions 
on  each  of  these,  and  we  present  them  in  turn. 

5.2.1.  Conceptual  issues 

One  of  our  major  concerns  was  to  understand  what  constitutes  an 
option,  and  how  this  depends  on  the  decision-maker  and  the  particular 
decision  problem.  In  general,  we  found  that  there  was  little  variation 
between  subjects  and  that  options  were  typically  very  brief.  A  common 
structure  for  the  design  of  options  was  to  liBt  several  elements  of  the 
problem,  two  or  three  approaches  to  solving  each  element,  and  then  to 
generate  a  string  of  options  by  permuting  these  solutions.  While  this 
led  to  significant  numbers  of  options,  this  kind  of  an  approach  does  not 
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lead  to  much  variability  or  differences  between  the  options  themselves. 
Thus,  for  all  of  the  subjects,  an  option  is  basically  a  combination  of 
primitive  action  alternatives. 

Perhaps  this  is  not  too  surprising  given  the  relatively  simple 
scenarios  we  used,  but  the  similarity  of  responses  is  quite  striking. 
The  main  differences  between  options  seems  to  be  one  of  detail.  Thus 
all  the  subjects  on  Day  1  realized  that  before  any  action  options  could 
be  generated  it  was  necessary  to  locate  the  B-52  and  determine  the 
status  of  the  equipment  and  crew.  There  were,  however,  many  different 
solutions  to  the  reconnaissance  problem:  long  range  photo-reconnaissance 
aircraft,  satellite  sensors,  Navy  task  force,  etc.  Similarly,  on  Day  2 
all  subjects  discussed  the  option  of  bombing  Small  Airport,  but 
described  different  combinations  of  equipment  and  munitions  for 
achieving  this. 

Although  there  was  considerable  consistency  about  the  nature  of  an 
option,  there  were  very  large  individual  variations  in  the  nimber  and 
quality  of  the  options  generated.  On  both  days,  these  ranged  from  the 
briefest  of  one-line  descriptions  to  relatively  detailed  analysis  of  the 
problem  and  possible  options.  Overall  though,  the  military  options 
tended  to  be  more  comprehensive  than  the  non-military  ones.  Our  general 
impression  is  that  there  is  as  much  variation  within  experimental  groups 
as  between  them.  We  attribute  this  to  individual  differences  in 
background  and  training,  rather  than  the  conditions  imposed  by  the 
experiment. 

The  overall  structure  of  the  option  sets  for  the  two  scenarios  was 
also  quite  different.  For  Scenario  1,  the  options  were  conditioned  upon 


the  supposed  fate  of  the  aircraft  and  the  key  to  generating  a  rich  set 
of  options  was  to  consider  many  possible  states-of-the-world.  For 
Scenario  2,  the  key  was  to  interpret  the  word  "neutralize"  as  broadly  as 
possible.  Thus  in  the  first  case,  creative  options  came  from  an  ability 
to  imagine  the  situation  in  a  variety  of  ways.  In  the  second  case  they 
came  from  flexibility  in  understanding  what  could  limit  the  enemy's 
activity. 

Although  difficult  to  quantify,  we  feel  that  the  options  generated 
for  Scenario  1  were  more  numerous  and  creative  than  those  generated  for 
Scenario  2.  We  believe  this  is  because  Scenario  1  was  well  outside  the 
experience  of  most  subjects.  causing  them  to  view  the  problem  as 
something  novel  and  requiring  ingenuity.  Scenario  2,  on  the  other  hand, 
seems  to  have  been  regarded  as  something  familiar,  so  that  fairly 
standard  solutions  could  be  applied. 

In  summary,  the  subjects  in  this  experiment  perceived  an  option  as 
a  combination  of  activities  designed  to  address  the  basic  parts  of  the 
overall  problem.  On  Day  1  particularly,  the  options  had  a  dominant  time 
dependent  element  and  took  on  the  character  of  a  plan.  There  appeared 
to  be  little  quantitative  difference  between  the  treatment  and  control 
groups  in  either  the  concept  of  an  option  or  the  level  at  which  it  was 
described.  The  variations  that  do  appear  are  probably  best  attributed 
to  individual  subject  differences  rather  than  the  experimental 


conditions. 
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5.. 2.2.  The  value  of  AHP 

Our  original  thinking  about  the  characteristics  of  an  aid  led  us  to 
believe  that  the  Analytic  Hierarchy  Process  had  features  that  would  be 
of  benefit  in  Option  Generation.  Having  developed  an  interactive 
computer-based  aid  that  allows  the  user  to  construct  and  prioritize  a 
hierarchy  of  decision  factors  for  the  problem,  one  of  our  goals  was  to 
determine  if  this  aid  would  be  of  benefit  to  the  subjects  in  our 
experiment. 

Unfortunately,  the  procedural  difficulties  that  the  subjects 
experienced  were  quite  severe.  On  Day  1  most  ran  out  of  time,  being  more 
concerned  with  the  mechanics  of  hierarchy  construction  and 
prioritization  than  with  the  actual  Option  Generation  task.  On  Day  2, 
we  attempted  to  alleviate  this  difficulty  by  conducting  a  group  exercise 
in  construction  and  prioritization.  However,  time  was  still  a 
significant  factor  and  forced  a  premature  termination  of  the 
prioritization.  Consequently,  our  comments  on  the  apparent  value  of  the 
AHP  aid  must  be  conditioned  on  this  important  constraint. 

If  we  compare  the  options  generated  by  the  control  and  treatment 
groups  on  both  days,  it  is  clear  that  control  group  responses  are 
generally  more  detailed  and  numerous.  We  believe  this  negative  result  to 
be  primarily  due  to  the  procedural  problems  (a  view  supported  by  our 
second  experiment,  see  Chapter  6)  and  so  have  not  attempted  a 
quantitative  analysis  of  the  options  generated.  Instead,  we  have  formed 
a  set  of  subjective  impressions  that  we  elaborate  below. 

There  is  some  evidence  that  the  range  of  options  generated  by  the 
treatment  group  on  Day  2  is  greater  than  that  of  the  control  group.  Our 
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subjective  interpretation  i6  that  Scenario  2  seemed  very  familiar, 
causing  the  control  group  to  think  mainly  in  terms  of  destroying  Small 
Airport.  The  treatment  group  more  often  considered  options  like 
blockades  and  further  diplomatic  activity,  prompted,  presumably,  by  the 
broader  view  of  the  problem  that  the  hierarchical  structuring  and 
prioritization  gives. 

These  tentative  conclusions  are  supported  by  a  reading  of  the 
questionnaires  returned  by  the  treatment  groups.  On  Day  1  the  responses 
had  a  negative  tone,  although  half  the  subjects  thought  that  the 
procedure  helped  them  to  be  more  analytic,  with  the  other  half  unsure. 
Most  subjects  on  Day  1  were  comfortable  with  the  pairwise  comparison 
technique,  but  some  commented  that  for  large  hierarchies  this  process 
becomes  very  tedious.  The  response  to  the  value  of  the  priorities  were 
very  mixed.  Several  subjects  were  unable  to  answer  because  they  did  not 
complete  the  prioritization.  Others  felt  them  useful  only  in  the  sense 
that  they  identified  the  most  dominant  factors.  That  is,  the  ordering 
was  more  important  than  the  absolute  values  of  the  weights.  Still 
others  said  they  ignored  them. 

On  Day  2,  the  subjects  felt  much  more  positive  towards  the  AHP 
process.  All  but  one  of  them  found  the  pairwise  comparison  process  to 
be  useful.  Several  also  found  the  priorities  to  be  of  value,  although 
many  more  were  either  unsure  or  offered  no  comment. 

On  balance  then,  the  structuring  aspects  of  the  AHP  aid  seems  to 
have  been  received  favorably  and  perceived  as  useful  by  the  subjects. 
The  value  of  priorities  seems  to  be  limited  however.  As  a  comment,  it 
is  characteristic  of  the  AHP  that  both  hierarchical  structuring  and 
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prioritization  have  to  be  completed  before  the  full  power  of  the  aid  can 
be  realized,  and  in  limited  time  decisions  it  may  not  be  possible  to  do 
this.  Sufficient  training  in  the  use  of  AHP  would  tend  to  alleviate,  but 
not  negate,  this  problem. 

There  is  little  evidence  to  suggest  that  creativity  is  enhanced  by 
AHP.  Indeed  several  subjects  made  comments  to  the  effect  that  while  the 
structuring  exercise  increases  understanding,  there  is  a  possibility 
that  the  hierarchy  restricts  creativity  by  forcing  the  decision  maker  to 
accept  a  static  relationship  between  the  decision  factors. 

In  summary,  the  data  generated  by  this  experiment  does  not  allow  us 
to  draw  any  quantitative  conclusions  about  AHP.  However,  we  can  see 
clearly  that  it  has  value  as  a  tool  for  analyzing  the  structure  of 
problems.  On  the  other  hand,  it  is  a  relatively  complex  procedure  and 
requires  the  decision  maker  to  commit  a  substantial  amount  of  effort 
before  its  value  can  be  realized.  A  more  flexible  and  robust  version  of 
the  software  would  help  alleviate  some  of  these  problems,  but  in 
restricted  time  situations  we  should  expect  its  effectiveness  to  be 
reduced. 


5.2.3.  Procedural  issues 


Perhaps  the  main  lesson  learned  from  this  experiment  is  that 
performing  an  experiment  that  attempts  to  assess  the  value  of  a  decision 
aid  in  a  complex  decision  situation  is  distinctly  non-trivial.  In  this 
section  we  will  highlight  seme  of  the  more  important  issues. 

First,  it  is  vital  that  the  subjects  practice  with  the  aid  before 
they  are  asked  to  use  it  under  experimental  conditions.  When  the  aid  is 


a  complex  computer  program,  there  is  a  significant  learning  period 
during  which  the  user  overcomes  his  preoccupation  with  the  procedural 
questions.  We  estimate  that  the  user  needs  several  exposures  to  AHP, 
both  the  theory  and  the  practice,  before  we  could  expect  him  to  be 
confident  enough  to  apply  it  to  a  real  problem,  in  such  a  way  that  he 
uses  most  of  his  energy  thinking  about  the  problem  rather  than  about 
using  the  tool. 

Second,  it  is  important  to  make  sure  that  the  subjects  understand 
exactly  what  is  required  of  them.  They  need  clear  instructions  and 
unambiguous  definitions.  Scenario  2,  in  particular,  caused  same 
confusion.  The  description,  we  thought,  clearly  emphasizes  that  the 
Wing  Commander  is  the  decision-maker  asked  to  generate  the  options.  Yet 
many  subjects  offered  options  available  only  at  a  much  higher  level  (for 
example,  pursuing  various  diplomatic  activities). 

Finally,  while  the  Debriefing  Questionnaire  was  well  structured  and 
allowed  us  to  compare  responses,  the  decision  to  allow  a  free  format  for 
the  options  themselves  considerably  hampered  our  ability  to  derive  any 
quantitative  results.  There  was  considerable  variation  in  style  and 
presentation,  making  the  analysis  of  individual  responses  very  time 
consuming  and  error  prone. 

5.3.  Summary 

In  our  search  for  ways  of  aiding  decision-makers  in  the  task  of 
Option  Generation,  we  identified  the  Analytic  Hierarchy  Process  as 
potentially  useful,  and  in  this  Chapter  we  have  described  an  experiment 
to  test  its  value.  Other  important  goals  were  to  refine  our 
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understanding  of  the  concept  of  an  option  and  to  assess  the  problems 
associated  with  performing  experiments  of  this  kind. 

The  data  we  collected  during  the  experiment  has  allowed  us  to  draw 
some  6trong  procedural  conclusions  but  rather  weak  theoretical  ones.  As 
a  result  we  have  only  been  able  to  assess  the  value  of  AHP  concept  in  a 
very  preliminary  way. 

Briefly,  the  lessons  are: 

(1)  Group  use  of  AHP  with  a  mediator  appears  more  satisfactory  than 
individual  use. 

(2)  Hierarchical  structuring  of  the  problem  does  help  problem 
understanding,  but  we  found  no  evidence  that  the  overall  AHP 
procedure  helps  generate  creative  options. 

(3)  An  option  is  a  combination  of  primitive  action  alternatives,  with 
time  dependence  of  actions  often  made  explicit. 

(4)  Individual  experience  appears  to  be  a  dominant  factor  in 

determining  the  richness  of  the  options  generated. 

(3)  It  is  most  important  that  subjects  are  given  adequate  time  to 
familiarize  themselves  with  the  aid. 

(6)  It  is  important  to  provide  clear  experimental  instructions. 

(7'  Assessment  of  free  form  written  responses  is  very  hard.  While 
there  is  a  danger  of  unduly  constraining  the  subjects,  more 
structured  responses  are  necessary  if  meaningful  quantitative 
analysis  is  to  be  performed. 


6.  THE  STANFORD  EXPERIMENT 


An  important  goal  of  this  experiment  was  to  test  a  simple  version 
of  our  Option  Prompting  Environment.  This  consisted  of  a  collection  of 
aids  from  which  subjects  were  free  to  select  as  many  or  as  few  as  they 
wished,  and  at  whatever  time  they  wished.  We  attempted  to  provide  a 
broad  selection  of  problem  solving  aids,  including  information  about  the 
problem,  aids  for  structuring  the  problem,  and  creativity  aids.  The 
information  packet  was  specific  to  the  problem,  whereas  the  structuring 
and  creativity  aids  were  quite  general.  The  structuring  aids  were 
designed  to  broaden  the  subjects'  conception  of  the  problem.  One  aid 
was  a  simple  checklist,  which  urged  subjects  to  consider  the  political, 
social  and  economic  factors  of  the  problem.  A  second  structuring  aid 
was  a  problem  solving  template,  which  required  subjects  to  generate  a 
matrix  of  issues  important  to  the  problem.  The  actual  template 

consisted  of  a  matrix  of  boxes,  to  be  filled  in  with  a  list  of  the 
actors,  each  actor's  goals,  resources,  and  constraints,  and  any  possible 
environmental  factors  which  would  figure  in  the  problem.  The  creativity 
aids  were  designed  to  help  the  subjects  "break  set"  (Luchins  [50]),  and 
shift  their  understanding  of  the  problem,  or  shift  to  a  new  area  of 
possible  solutions.  Included  in  the  creativity  packet  was  an  exercise 
which  asked  subjects  to  list  the  assumptions  they  had  about  the  problem, 
and  write  down  a  negation  of  the  assumption.  Another  exercise  listed  a 
variety  of  "brainstorming"  questions  (Osborne  [57);  Gordon  [26]).  A 
third  exercise  listed  a  set  of  unusual  words,  and  asked  for  the 

subject's  free  associations  to  those  words.  A  subject  who  had  spent 
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some  time  generating  very  similar  options  would  be  forced  to  use  a 
different  area  in  semantic  memory  during  the  free  association  exercise, 
and  therefore  think  of  solutions  slanted  tcward  a  different  aspect  of 
the  problem. 

We  also  re-tested  the  effectiveness  of  the  Analytic  Hierarchy 
Process  (AHP).  We  hypothesized  that  the  computer  implementation  of  AHP 
would  motivate  subjects  to  develop  a  broad  conception  of  the  problem, 
and  to  understand  the  relationships  among  elements  of  the  problem.  Such 
an  understanding  appears  particularly  important  for  open  problems,  in 
which  the  problem  structure  is  neither  obvious  nor  predefined. 

A  major  difficulty  of  evaluating  problem  solving  and  creativity 
aids  is  the  issue  of  measurement.  The  concept  of  a  good  option  or  a 
good  set  of  options  is  ill-defined,  and  may  vary  across  situations. 
Thus,  an  important  aspect  of  the  experiment  was  the  development  of  a  set 
of  measures  of  option  generation  performance. 

We  hypothesized  that  cognitive  style  might  mediate  the  usefulness 
of  problem  solving  aids.  For  example,  same  people  may  already  possess 
effective  strategies  for  problem  solving.  External  problem-solving  aids 
might  hinder  the  use  of  these  subjects'  own  strategies,  and  produce 
poorer  performance.  On  the  other  hand,  people  who  are  relatively 
uncreative,  or  who  have  difficulty  structuring  problems  might  benefit 
more  from  our  problem-solving  aids.  We  used  two  tests  of  cognitive 
style  in  an  exploratory  manner.  The  first  was  the  Embedded  Figures  Test 
[84],  which  divides  subjects  into  field  dependent  versus  field 
independent.  Field  independence  versus  dependence  maps  onto  the 
divergent  versus  convergent  thinking  dimension.  People  who  are  field 
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independent  tend  to  be  divergent  thinkers,  which  is  supposedly  typified 
by  fluid  and  flexible  thinking.  Field  dependent  people  tend  to  be 
higher  on  convergent  thinking.  Therefore,  the  problem  solving  aids  may 
be  more  useful  for  the  field  dependent  subjects.  The  second  test  of 
cognitive  style  was  the  attention  subtest  of  the  Test  of  Attentional  and 
Interpersonal  Style  (Niedeffer  [56])  which  measures  variations  in  the 
direction  and  general  breadth  of  attention. 

6..  1_.  Experimental  Method 

Thirty  Stanford  University  students  were  obtained  as  subjects 
through  signs  requesting  volunteers  for  an  experiment  in  problem 
solving. 

We  wished  to  test  the  problem  solving  aids  on  an  open,  real-life 
problem.  Furthermore,  we  wanted  to  choose  a  problem  which  subjects 
would  be  motivated  to  solve,  and  about  which  they  had  a  large  store  of 
domain  relevant  knowledge.  (Perceived  lack  of  domain  knowledge  has  been 
indicated  as  a  block  to  options  generation  in  comments  by  several 
subjects  in  the  Monterey  experiment.)  After  pre-te6ting  several 
problems,  we  chose  the  problem  of  campus  security  in  response  to  a 
recent  series  of  attacks  on  women  at  Stanford.  Since  all  subjects  were 
students  on  the  Stanford  campus,  they  had  a  great  deal  of  information 
about  the  physical  setting,  the  people  involved,  and  their  various 
goals,  resources,  and  constraints.  The  attacks  were  an  important  and 
visible  issue  on  campus  at  the  time  the  experiment  was  run,  and  subjects 
appeared  to  be  highly  motivated  when  generating  options. 


The  ten  subjects  in  the  control  condition  worked  on  the  problem 


unaided.  The  ten  subjects  in  the  Simple  Environment  condition  were 
provided  with  a  packet  of  information  about  the  problem,  a  set  of 
structuring  tools,  and  a  set  of  creativity  tools.  The  ten  subjects  in 
the  AHP  condition  were  first  trained  in  the  use  of  a  computer  aided 
version  of  the  Analytic  Hierarchy  Process,  and  then  were  sllowed  to  use 
AHP  while  working  on  the  problem.  Subjects  were  tested  individually. 

All  subjects  read  a  description  of  the  problem,  including 
information  about  the  attacks,  and  steps  the  university  had  taken  in 
response.  Subjects  were  asked  to  play  the  role  of  a  member  of  a 
university  advisory  committee,  and  attempt  to  generate  solutions  to  the 
problem.  Subjects  then  wrote  a  description  of  the  problem  in  their  own 
words.  They  were  then  asked  to  list  all  initial  options,  which  were 
defined  as  ideas  they  had  about  the  problem  before  entering  the 
experiment.  Finally,  subjects  were  given  a  maximum  of  two  and  a  half 
hours  to  generate  new  options  to  the  problem.  They  were  instructed  to 
list  every  idea  which  occurred  to  them,  without  editing  ideas  which  they 
thought  were  "silly"  or  infeasible.  Subjects  were  allowed  to  stop  the 
experiment  at  any  point. 

6_.  1_.  1_.  AHP  Condition 

One  or  two  days  prior  to  the  experiment,  people  in  the  AHP  session 
were  given  a  three  hour  training  session  on  a  computer  implementation  of 
the  Analytic  Hierarchy  Process.  Each  subject  heard  a  short  lecture- 
demonstration,  and  used  AHP  while  working  on  a  practice  problem.  During 
the  actual  experiment,  AHP  subjects  were  given  access  to  AHP,  with 
instructions  to  use  it  to  help  generate  new  options.  The  instructions 


stressed  that  the  main  goal  of  the  experimental  session  was  to  generate 
new  options  regarding  the  problem,  rather  than  focusing  on  an  exhaustive 
use  of  the  AH?  program. 


6. 1_.  2.  Simple  Environment  Condition 

People  in  the  Simple  Environnent  condition  were  provided  with  three 
different  aids,  an  information  packet,  a  structuring  packet,  and  a 
creativity  packet.  The  information  packet  contained  a  copy  of  a 
newspaper  interview  with  two  of  the  victims,  a  letter  from  the  chief  of 
police  to  the  community  detailing  the  University  response  to  the 
attacks,  a  description  of  the  campus  escort  service  including  hours  of 
operation,  and  a  map  of  the  campus.  The  structuring  packet  contained  a 
simple  checklist,  which  reminded  subjects  to  consider  the  economic, 
political,  and  social  aspects  of  the  problem.  A  second  structuring  aid 
was  the  template.  The  template  consisted  of  a  5  by  7  matrix  of  boxes 
labeled  "Actors",  "Goals", "Resources",  "Constraints",  and  "Environmental 
Factors".  An  example  of  a  completed  template  for  a  different  problem 
(eradication  of  the  Mediterranean  Fruit  Fly  in  California)  was  included 
as  a  guide.  The  creativity  packet  contained  three  exercises.  The  first 
was  a  list  of  "brainstorming"  questions,  similar  to  those  described  in 
Gordon' 8  book,  Svnectics.  The  second  exercise  was  designed  to  encourage 
free  association.  The  free  association  exercise  provided  subjects  with 
a  list  of  20  words,  and  asked  them  to  write  down  the  first  word  which 
occurred  to  them  as  they  read  each  one.  The  challenge  assumptions 
protocol  asked  people  to  write  down  any  assumptions  they  had  about  the 
problem,  and  then  write  down  the  negation  of  their  assumption,  stated  in 
a  positive  manner. 
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6_.  1_.3.  Control  Condition 

Subjects  in  the  Control  Condition  were  simply  provided  with 
worksheets  for  recording  their  options.  They  were  not  provided  with  any 
problem  solving  aids. 

6_.  1_. 4.  Post-Experimental  Tests 

After  giving  the  instructions,  the  experimenter  left  the  room.  The 
subjects  were  allowed  to  work  on  the  problem  until  they  felt  they  had 
exhausted  their  options,  or  for  a  maximum  of  two  and  a  half  hours.  At 
the  end  of  the  session,  the  experimenter  returned  and  gave  each  subject 
two  tests  of  cognitive  style.  The  Attention  Subtest  of  the  Test  of 
Attentional  and  Interpersonal  Style  (TAIS)  consisted  of  75  multiple 
choice  questions  designed  to  divide  subjects  along  two  dimensions: 
direction  of  attention  (internal  versus  external)  and  breadth  of 
attention  (broad  versus  narrow  focus).  The  Embedded  Figures  Test 
consists  of  12  geometric  figures  with  simpler  figures  embedded  in  them. 
Subjects  are  timed  as  they  attempt  to  find  and  trace  the  embedded 
figure.  People  who  attain  low  scores  of  total  time  are  relatively 
field-independent.  They  readily  separate  objects  or  ideas  from  the 
surrounding  context.  People  with  high  Bcores  on  the  &nbedded  Figures 
Test  are  relatively  field-dependent,  and  have  difficulty  separating 
objects  or  ideas  from  the  background  in  which  they  are  embedded.  After 
the  cognitive  style  tests,  subjects  in  the  AHP  condition  and  Simple 
Environment  condition  were  given  a  questionnaire  about  the  problem 
solving  aids.  Finally,  all  subjects  were  debriefed  as  to  the  nature  of 
the  experiment. 
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6.1_.5..  Evaluation  of  Options 

The  solutions  to  the  problems  were  evaluated  along  a  nunber  of 
dimensions.  The  first  dimension  was  the  nunber  of  options  generated  by 
each  person.  A  second  dimension  was  the  breadth  of  those  options. 
Based  on  pilot  work  with  the  problem,  a  list  of  13  mutually  exclusive, 
comprehensive  categories  of  options  was  devised  (see  Table  6.1).  For 
example,  one  category  included  anything  which  required  physical  changes 
to  campus,  another  category  included  all  suggestions  of  care  for 
victims,  and  a  third  category  consisted  of  all  options  which  related  to 
escorts.  The  options  for  each  subject  were  sorted  into  the  13 
categories  by  a  judge  who  was  blind  to  the  subject's  condition,  and  the 
number  of  categories  which  contained  an  entry  constituted  the  subject's 
breadth  score. 

Each  option  was  also  rated  in  terms  of  novelty,  feasibility  and 
benefit.  Novelty  was  defined  as  the  degree  to  which  an  option  was  new 
or  unusual.  Feasibility  was  defined  as  the  combined  dollar  and 
psychological  costs  of  implementing  an  option.  Benefit  was  defined  as 
the  degree  to  which  an  option  would  alleviate  the  overall  problem.  A 
comprehensive  list  of  all  options  generated  by  subjects  in  the 
experiment  was  assembled.  Four  judges  then  rated  each  option  on  each  of 
the  three  dimensions.  The  judges  used  a  separate  0  to  100  scale  for 
novelty,  feasibility,  and  benefit.  Two  anchor  points  for  each  scale 
were  chosen  by  consensus  prior  to  the  judges  assessment  of  all  the 
options.  The  mean  score  of  the  four  judges  was  computed  for  each 
option's  novelty,  feasibility,  and  benefit  score.  Then  the  total 
novelty,  feasibility  and  benefit  scores  for  each  subject  was  computed  by 
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1.  Victim  Care 


2.  Personal  Self-defense 

3.  Classes  in  Self-defense 

4.  Personal  publicity  and  education 
3.  Group  publicity  and  education 

6.  Physical  Changes  to  campus 

7.  Rules  of  behavior  or  restrictions 

8.  Escorts 

9.  Victim  Counseling 

10.  Society:  Changes  in  law 

11.  Society:  Changes  in  attitudes 

12.  Police  Action 

13.  Other 


Table  6.1  Categories  of  Options 
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adding  the  scores  of  all  the  options  in  that  subject's  options  set. 
Finally,  the  average  novelty,  feasibility  and  benefit  score  was  computed 
for  each  subject. 

6.2_.  Results 

A  separate  one-way  analysis  of  variance  was  performed  on  each  of 
the  measures:  number  of  options,  breadth,  novelty,  feasibility,  and 
benefit.  The  results  are  shown  in  Table  6.2.  Subjects  in  the  Simple 
Environment  and  AHP  conditions  did  not  differ  from  the  control  group  in 
the  average  benefit,  feasibility  or  novelty  of  the  options  they 
generated.  There  was  also  no  difference  between  the  treatment 
conditions  and  the  control  group  in  the  ntmber  of  options  generated. 
However,  both  the  treatment  groups  generated  a  greater  breadth  of 
options  than  did  the  control  group  subjects.  Subjects  could  generate 
options  in  as  many  as  13  categories.  Subjects  in  the  AHP  group  had  an 
average  breadth  score  of  8.7  categories,  subjects  in  the  Simple 
Environment  group  had  an  average  breadth  score  of  7.7  categories, 
whereas  control  group  subjects  had  an  average  breadth  score  of  6.2 
categories  (F[2,27]*6.92,  p<0.005).  Separate  t-te6t  comparisons 
revealed  that  while  the  AHP  group  and  the  Simple  Environment  group  were 
not  significantly  different  in  terms  of  the  breadth  scores,  both  were 
significantly  different  from  the  control  group. 

Subjects  were  free  to  spend  up  to  130  minutes  on  the  problem. 
Subjects  differed  significantly  in  the  time  spent  on  the  problem  across 
the  three  conditions  (F[2,27]"3.60,  p<0.05).  Subjects  in  the  AHP 
condition  spent  significantly  more  time  on  the  problem  (1>4. 5  minutes) 
than  subjects  in  either  the  Simple  Envirotment  condition  (106.2)  or  the 
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DEPENDENT  MEASURE 

GROUP 

NUMBER 

8READTH 

NOVELTY 

FEASIBILITY 

BENEFIT 

TIME 

SIMPLE 

ENVIRONMENT 

17.0 

7.7 

23.11 

50.14 

29.42 

106.2 

AHP  GROUP 

19.0 

8.7 

24.92 

44.70 

30.54 

134.5 

CONTROL 

GROUP 

15.2 

6.2 

23.79 

54.09 

28.19 

99.5 

F  STATISTIC 

2,  27  df 

F*  .59 

F«6.92 

p  <  .01 

F*,19 

F*1.89 

F«.25 

F*3.60 

p  <  .05 

Table  6.2  Statistics  for  Treatments 
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control  condition  (99.5).  The  difference  may  be  due  simply  to  the  time 
consuming  nature  of  the  AHP  procedure.  In  that  case,  however,  it  is 
interesting  that  working  through  the  Analytic  Hierarchy  Process,  or 
using  any  of  the  packets  in  the  Simple  Environment  conditions  did  not 
hinder  the  performance  of  subjects  in  those  conditions.  An  alternative 
possibility  is  that  the  treatments  may  have  made  the  problem  session 
more  interesting  or  perhaps  less  frustrating  relative  to  the  experience 
of  the  control  group,  and  thus  subjects  were  more  willing  to  work  on  the 
problem. 

Figures  6.1  -  6.5  are  histograms  of  subject  averaged  scores  on  each 
of  our  five  measures  of  "goodness":  nunber,  breadth,  novelty, 
feasibility  and  benefit.  The  vertical  axes  show  the  upper  end  points  of 
each  interval  used  in  constructing  the  histograms.  Thus  in  Figure  6.1  we 
can  see  that  only  one  subject  in  the  Simple  Environment  group  generated 
less  than  or  equal  to  9.0  options,  and  three  generated  more  then  9.0  but 
less  than  or  equal  to  12.0.  Since  we  have  so  few  subjects  in  each  group 
the  comments  that  follow  on  the  differences  between  histograms  are 
necessarily  qualitative  and  should  be  treated  with  some  caution.  We  are 
looking  for  effects  that  would  be  worth  exploring  in  a  more  extensive 
experiment.  All  the  histograms  show  substantial  variation  within  groups 
emphasizing  that  there  are  large  differences  within  the  subject 
population. 

The  number  scores  are  shown  in  Figure  6.1.  Notice  that  all  three 
groups  have  most  subjects  generating  about  15  options  with  just  a  few 
generating  significantly  more.  The  breadth  scores  are  shown  in  Figure 
6.2  and  reflect  the  overall  group  differences  discussed  above.  As  shown 
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Figure  6.1.  Histograms  for  NUMBER  scores 
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Figure  6.2  Histograms  for  BREADTH  scores 
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in  Figure  6.3,  the  two  treatment  groups  show  a  restricted  range  of 
novelty  relative  to  the  control  group.  In  fact,  the  control  group 
exhibits  approximately  twice  the  standard  deviation  of  either  the  AHP 
group  or  the  Simple  Envirormen t  group,  although  there  is  no  difference 
among  the  three  groups  in  terms  of  average  novelty  score.  The  treatment 
groups  may  have  eliminated  extremely  low  average  novelty  scores,  as  well 
as  extremely  high  average  novelty  scores.  The  feasibility  histograms  in 
Figure  6.4  show  a  skewed  distribution  for  the  AHP  group.  There  actually 
was  a  trend  toward  less  feasible  options  among  the  AHP  subjects 
(mean=44.7)  relative  to  the  control  subjects  (mean* 54. 1) .  The  AHP 
process  seems  to  lead  to  a  fairly  automatic  process  of  crossing 
dimensions  and  exploring  the  problem  space.  For  example,  one  subject 
considered  the  actors  in  the  problem  (students.  University,  University 
Police,  local  community,  local  community  police,  etc.)  and  applied  every 
option  generated  for  one  to  all  the  actors.  He  first  suggested  that  the 
University  sponsor  self-defense  classes,  a  quite  feasible  option,  but 
followed  that  suggestion  with  the  less  feasible  options  of  requiring  the 
University  Police,  the  local  community,  and  the  local  community  police 
to  sponsor  self-defense  classes.  Such  an  approach  may  lead  to 
infeasible  suggestions  which  would  be  edited  out  as  "nonoptions"  by 
members  of  the  control  group.  Finally,  the  benefit  scores  shown  in 
Figure  6.5  also  hint  at  some  differences  between  the  groups.  The  effect 
appears  to  be  one  of  reducing  the  within  group  variability  as  we  move 
from  the  Simple  Environment  to  AHP  to  the  Control  group. 


6.2.1.  Interaction  of  Treatments  and  Cognitive  Style 

The  interaction  of  field  dependence  and  independence  (as  measured 
by  the  Embedded  Figures  Test)  with  the  treatments  was  assessed  by 
dividing  the  subjects  in  each  condition  into  two  groups,  Field  Dependent 
(above  the  median  in  total  response  time)  and  Field  Independent  (below 
the  median  in  total  response  timed).  The  data  was  then  analyzed  via  a 
two-way  (Field  dependence  by  condition)  analysis  of  variance  for  each  of 
the  5  option  measures:  number,  breadth,  novelty,  feasibility,  and 
benefit.  Field  dependence  did  not  interact  with  the  effectiveness  of 
the  treatment  groups  for  any  of  the  measures  (see  Table  6.3).  However, 
field  dependence  significantly  affected  the  number  of  options  generated 
(F[l,24]=6.66,  p<0.05).  The  subjects  who  had  low  scores  on  the  Embedded 
Figures  Test,  and  thus  were  relatively  field  independent  tended  to 
generate  more  options  than  subjects  with  high  scores  on  the  Bnbedded 
Figures  Test.  However,  the  groups  of  low  score  subjects  have 
approximately  twice  the  variance  of  people  with  high  scores.  Although 
the  analysis  of  variance  is  quite  robust  with  respect  to  violations  of 
homogeneity  of  variance,  we  must  interpret  this  difference  with  caution. 
We  did  not  perform  a  similar  analysis  of  the  interaction  of  attentional 
style  with  measures  of  option  generation,  because  our  subjects  varied 
little  on  the  subscores  of  the  TAIS.  Our  population  of  undergraduate 
and  graduate  students  may  represent  a  narrow  range  of  attentional 
styles. 

6..  3.  Discussion 

The  two  option  generation  aids  tested  in  this  experiment,  the 
Analytic  Hierarchy  Process  and  the  combination  of  information. 
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SIMPLE 

ENVIRONMENT 

AHP 

CONTROL 

F-STATISTICS 

HIGH 

LOW 

HIGH 

LOW 

HIGH 

LOW 

INTERACTION 
df  2,  24 

DEPENDENCE 
df  1,  24 

NUMBER 

14.8 

19.2 

13.6 

24.4 

12.6 

17.1 

.58 

6.66 

p  <  0.05 

BREADTH 

D 

7.8 

8.2 

9.2 

5.6 

6.8 

.03 

2.07 

NOVELTY 

22.9 

23.3 

24.0 

25.8 

22.7 

24.9 

.05 

.34 

FEASIBILITY 

51.9 

48.4 

46.7 

42.7 

55.6 

52.5 

.00 

.72 

BENEFIT 

28.9 

29.9 

29.8 

31.3 

28.1 

28.3 

.02 

.10 
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structuring,  and  creativity  aids  in  the  Simple  Environment  condition 
significantly  increased  the  range  or  breadth  of  options  subjects 
generated.  However,  the  two  treatment  groups  did  not  differ  from  the 
control  group  in  the  number  of  options,  or  average  novelty,  feasibility, 
or  benefit  of  their  options.  Thus,  there  did  not  seem  to  be  a  trade-off 
of  increasing  breadth  and  decreasing  number  or  quality  of  options 
generated. 

Both  the  Analytic  Hierarchy  Process  and  the  Simple  Environnent 
condition  contain  a  variety  of  components  which  may  have  resulted  in  the 
subjects  producing  a  broader  range  of  options.  Gettys  and  Fisher  [23] 
proposed  that  the  information  used  in  generating  a  structure  for  a 
problem  is  retrieved  from  memory  through  a  multistage  search  process. 
AHP  is  a  procedure  which  requires  the  subject  to  formally  represent  the 
structure  of  the  problem.  The  AHP  program  continues  prompting  subjects 
for  more  entries  at  each  level  of  the  hierarchy,  and  for  more  levels. 
Such  prompting  may  increase  the  time  subjects  spend  searching  for 
aspects  of  the  problem,  and  increase  the  number  of  elements  in  their 
problem  representation.  Therefore,  they  may  later  attempt  to  generate 
options  in  a  broader  range  of  categories.  The  structuring  template  in 
the  Simple  Environment  condition  would  have  a  similar  effect.  The 
matrix  presented  was  large;  5  categories  (actors,  goals,  resources, 
constraints  and  environmental  factors),  by  7  possible  examples  of  each 
category.  Subjects  attempted  to  completely  fill  in  the  matrix,  which 
again  may  have  prompted  a  broad  view  of  the  problem.  Future  research 
might  center  on  pinpointing  the  source  of  the  effect  on  breadth  scores. 
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The  AHP  and  Simple  Enviroment  treatments  did  not  increase  the 
number  of  options  generated,  nor  their  average  novelty,  feasibility,  or 
benefit.  The  experiment  should  be  regarded  as  a  pilot  study,  because  of 
the  small  nunbers  of  subjects  tested.  However,  new  forms  of  the  aids 
and  new  aids  aimed  specifically  at  fluency,  novelty,  feasibility,  or 
benefit  of  options  should  be  explored.  The  type  of  options  generated 
may  depend  on  the  subjects  interpretation  of  the  experimenter's  goals. 
For  example,  Hudson  [38]  studied  creativity  in  schoolboys.  He  asked 
them  to  generate  uses  for  common  objects,  first  role-playing  a  serious 
engineer,  second  as  a  bohemian  artist.  The  latter  instructions  resulted 
in  far  more  novel  responses  than  the  former.  Hudson  suggests  that 
everyone  has  the  capacity  to  be  creativity,  if  they  believe  the 
situation  calls  for  creativity.  We  asked  subjects  to  role-play  being  a 
member  of  a  university  advisory  committee.  Those  instructions  may  have 
biased  the  subjects  toward  particular  types  of  options.  One  aid  might 
be  to  specifically  ask  subjects  to  play  a  variety  of  different  roles. 

The  effectiveness  of  problem-solving  aids  may  interact  with  the 
type  of  problem  to  which  they  are  applied.  Although  no  formal  taxonomy 
of  problems  exists,  we  acknowledged  the  difference  between  open  problems 
(broad  problems  for  which  there  is  no  single  right  answer)  and  closed 
problems  (problems  with  one  clearly  defined  solution).  Another 
distinction  among  problems  may  be  drawn  with  regard  to  the  type  of 
solutions  which  are  applicable.  In  the  first  experiment  of  this 
contract,  subjects  were  asked  to  generate  solutions  to  the  problem  of  a 
downed  aircraft.  Subjects  responded  with  lengthy  plans  to  find  and 
retrieve  the  craft.  Each  "option"  actually  consisted  of  a  series  of 
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steps  in  the  plan.  In  the  present  experiment,  options  tended 
single  steps.  A  single  option  might  contribute  partially  to  the 
solution  of  the  problem,  or  it  might  solve  the  problem  entirely.  The 
options  did  not  serially  depend  on  one  another,  and  in  fact  could  be 
invoked  simultaneously.  Such  differences  in  the  type  of  solution 
(involved  plans  versus  a  collection  of  single  options)  required  by  a 
problem  may  determine  which  types  of  aids  are  best  suited  for  a  problem. 

We  measured  "goodness"  of  option  sets  in  a  number  of  ways.  Option 

fluency  was  scored  simply  by  counting  the  number  of  options.  Novelty, 
feasibility,  and  benefit  were  scored  by  the  average  subjective 

weightings  of  four  judges.  Breadth  was  determined  by  devising  a  set  of 
categories  of  options  on  the  basis  of  pretests.  Although  we  feel  these 
measures  tap  into  several  dimensions  of  intuitive  notions  of  "good" 
solutions,  they  may  be  evaluated  in  other  ways.  For  example,  Davis  [14] 
suggests  measuring  novelty  on  the  basis  of  statistical  frequency.  An 
option  which  occurs  just  once  in  the  total  pool  of  options  would  receive 
a  higher  novelty  score  than  an  option  which  occurs  two  or  more  times. 
In  the  present  experiment,  we  averaged  the  subjects'  novelty, 

feasibility  and  benefit  scores.  An  alternative  would  be  to  count  the 
number  of  options  which  exceed  a  certain  cutoff  point  on  novelty, 
feasibility,  or  benefit,  or  to  compute  the  proportion  of  options  which 
exceed  that  cutoff  point.  When  time  is  a  factor,  the  rate  of  options 
generated,  or  the  rate  of  novel  options  generated,  may  be  computed.  The 
choice  of  measures  may  be  guided  by  an  a  priori  conception  of  what  will 
be  relevant  for  applications  of  the  problem  solving  aid.  If  people  will 
use  the  problem-solving  aid  primarily  under  time  pressure,  rate  measures 


became  more  important  during  testing.  Alternatively,  if  the  aid  will  be 
used  in  an  enviroment  which  weighs  feasibility  heavily,  then  that 
dimension  should  be  stressed  during  testing. 

Finally,  the  post-test  questionnaires  completed  by  the  subjects  in 
the  Simple  Enviroment  and  AHP  groups  give  us  same  additional  insights. 
As  we  would  expect  from  our  statistical  results,  there  was  significant 
variation  within  the  treatment  groups.  Most  subjects  in  the  Simple 
Environment  treatment  attempted  to  use  all  the  tools  we  provided.  They 
often  did  this  even  when  they  felt  the  tools  were  of  no  particular 
benefit  and  despite  our  instructions  to  the  contrary.  Many  found  the 
word  association  tool  to  be  superfluous  and  the  brainstorming  questions 
to  be  unhelpful.  However,  there  were  several  strong  positive  responses 
to  both  the  structuring  template  and  the  challenge  assumptions  protocol. 
A  number  commented  that  they  would  have  benefitted  from  having  a  friend 
around  to  test  their  id«.  s  against.  We  might  interpret  this  as 
indicating  the  need  for  a  simple  option  evaluation  aid,  something  we  did 
not  provide. 

Subjects  in  the  AHP  treatment  most  often  commented  on  the  program's 
lack  of  flexibility  and  robustness,  reinforcing  the  lessons  learned  at 
Monterey.  Several  obviously  found  it  hard  to  organize  their  thinking 
around  the  structure  imposed  by  AHP,  but  others  seemed  to  find  it 
helpful  and  constructed  elaborate  hierarchies.  Perhaps  the  most 
interesting  reaction  that  we  observed  is  that  the  majority  of  subjects 
used  AHP  as  an  option  evaluation  tool  rather  than  a  problem  structuring 
tool.  That  is,  they  first  generated  options  unaided  and  then  used  AHP 
to  construct  a  value  model  against  which  to  test  the  options.  We  had 


expected  that  they  would  construct  a  prioritized  hierarchy  of  decision 
factors  and  then  use  this  to  help  them  generate  options. 

Such  variation  in  response  to  the  treatments  emphasizes  the  need  to 
provide  a  selection  of  tools.  Future  experiments  should  include  a  wider 
choice  of  tools  and  should  be  designed  so  that  frequency  and  length  of 
tool  use  can  be  recorded.  This  data  would  be  extremely  valuable  in 
helping  build  a  picture  of  typical  decision  behaviors,  and  would  help 
refine  the  content  and  form  of  the  toolbox. 
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7.  SUMMARY  AND  CONCLUSIONS 


In  this  final  chapter  we  summarize  the  research  performed  during 
the  contract,  discuss  possible  extensions  of  the  ideas  and  provide 
insight  into  the  likely  impact  of  an  operational  Option  Prompting 
Environnent  on  Air  Force  decision  making. 

]_.  1_.  Project  Status 

The  purpose  of  the  effort  described  in  this  report  was  to  define 
and  explore  techniques  for  helping  decision  makers  generate  options  in 
decision  problems.  In  performing  this  research  we  completed  a  variety 
of  tasks: 

(1)  We  prepared  an  extensive  bibliography  of  appropriate  literature 
together  with  a  commentary  on.  the  key  publications. 

(2)  We  developed  a  computer  implementation  of  the  Analytic  Hierarchy 
Process. 

(3)  We  performed  an  experiment  designed  to  test  the  effectiveness  of 
AHP  as  an  option  generation  aid. 

(4)  We  devised  the  concept  of  an  Option  Prompting  Environnent  as  a 
fundamental  component  in  the  solution  of  the  option  generation 
problem. 

(5)  We  performed  a  second  experiment  designed  to  test  a  simple  "paper" 
version  of  the  environment,  at  the  same  time  re-testing  the  AHP 
aid. 
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Our  main  results  and  conclusions  fall  into  two  broad  categories: 
those  that  are  conceptual,  and  those  that  are  methodological.  Thus  part 
of  our  effort  has  been  to  define  the  problem  of  option  generation  and 
what  it  means  to  aid  the  decision  maker  in  this  task,  and  part  has  been 
to  define  and  use  procedures  for  experimentally  evaluating  our  ideas. 

The  conceptual  issues  may  be  summarized  as: 

(1)  The  task  of  option  generation  is  not  one  that  exists  in  isolation. 
Instead,  it  is  at  the  core  of  decision  making,  which  is  an 
iterative  and  dynamic  process.  We  should  not,  therefore,  treat 
option  generation  independently,  but  have  to  concern  ourselves 
with  the  overall  problem  of  decision  making. 

(2)  Option  generation  itself  is  a  creative  activity  that  cannot  be 
automated  for  the  "open"  problems  in  which  we  are  interested.  What 
i6  required  are  methods  for  enhancing  the  ability  of  the  decision 
maker  to  "break  set"  and  thereby  overcome  his  inherent  conceptual 
blocks. 

(3)  There  are  a  variety  of  reasons  why  options  do  not  get  generated. 
These  are  related  to  the  various  stages  of  the  decision  process  and 
the  corresponding  cognitive  limitations  of  human  decision  makers. 
In  general  it  is  not  sufficient  to  provide  only  a  single  options 
prompting  aid.  Rather  the  decision  maker  needs  an  appropriate 
environment  containing  several  aids  that  can  be  selected  and  used 
according  to  the  specific  decision  phases  in  which  the  decision 
maker  wants  or  needs  support. 
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(4)  The  enviroment  itself  can  be  simple  or  complex.  At  its  simplest, 
it  merely  provides  a  set  of  tools  and  rules  for  their  use  (a 
toolbox),  and  procedures  for  recording  options.  As  it  becomes  more 
complex,  it  incorporates  explicit  knowledge  about  the  decision 
making  process  and  the  limitations  of  human  decision  makers.  It 
could  serve  either  as  an  operational  aid  or  as  a  training 
environment. 

(5)  What  constitutes  an  option  depends,  at  least  in  part,  on  the  type 
of  problem.  For  the  Downed  B-52  scenario  in  the  Monterey 
experiment,  an  option  has  the  character  of  a  plan,  with  the  set  of 
options  consisting  of  alternative  plans.  In  the  campus  security 
problem,  however,  an  option  is  a  single  step  that  might  be  taken, 
either  alone  or  in  conjunction  with  others  in  the  option  set. 

(6)  Performing  the  AHP  provides  a  useful  structure,  indicates  the 
breadth  of  decision  factors  to  consider  and  the  priorities  of  these 
factors. 

Our  comments  on  the  methodological  issues  may  be  summarized  as: 

(1)  There  is  a  severe  measurement  problem,  both  in  defining  appropriate 
measures  of  effectiveness  and  in  making  them  operational.  In  our 
experiments  we  asked  subjects  to  generate  as  many  creative  options 
as  possible,  leaving  aside  the  question  of  whether  this  is  always 
the  most  appropriate  approach.  We  are  concerned  with  aspects  of  the 
set  of  options  as  well  as  with  the  individual  options  themselves. 
That  is,  we  are  interested  in  the  nunber  and  breadth  of  options 
generated  not  just  the  goodness  of  the  individual  options.  The 
development  of  consistent  valid  measuring  instruments  for 


subjective  concepts  such  as  novelty,  feasibility  and  benefit  is 
non-trivial . 

(2)  Computerized  aids  present  a  special  set  of  difficulties  when  used 
experimentally.  There  are  difficult  questions  that  remain  to  be 
answered  about  the  proper  design  of  the  man-machine  interface  and 
the  general  acceptibi lity  of  such  tools.  Mot  only  may  the  user  be 
unfamiliar  with  the  content  of  the  aid,  but  he  may  also  be 
unfamiliar  with  the  general  concept  (and  limitations)  of  computer- 
based  aids.  Experimental  evaluation  of  the  concept  of  an  aid  (eg. 
AHP)  cannot  be  separated  from  its  embodiment  in  the  computer  and 
the  procedures  to  use  it.  It  is  essential,  therefore,  to  provide 
adequate  training. 

(3)  Performing  formal  experiments  with  decision  aids  is  complicated  by 

the  complexity  of  human  decision  making  behavior.  In  order  that  the 
aid  receives  a  realistic  test,  we  need  to  provide  real-world 
problems.  This  makes  it  hard  to  control  the  independent  variables 
in  the  experiment  and  so  conduct  a  rigorous  test  of  the 

experimental  hypothesis.  This  is  a  major  methodological  issue  that 
is  not  resolved. 

Finally,  our  pilot  experiments  provide  support  for  our  contention 
that  both  the  AHP  and  an  option  prompting  environment  can  aid  the 
decision  maker  in  generating  options.  However,  the  effects  observed  are 
small.  The  Monterey  experiment  provided  us  with  some  qualitative 
understanding  of  the  issues  and  in  the  Stanford  experiment  we  were  able 
to  test  these  issues  more  formally. 
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2.. 2.  Future  Research 

This  has  been  a  basic  research  effort  to  determine  the  nature  of 
the  option  generation  problem.  In  exploring  the  many  issues  that  have 
arisen  ve  have  been  able  to  define  a  series  of  questions  that  should  be 
considered  in  any  future  research.  In  particular  we  believe  that  an 
implementation  of  the  Option  Prompting  Environment  would  be  an 
appropriate  next  step.  The  crucial  elements  of  this  are  the  knowledge 
base  and  the  toolbox  of  aids.  Without  the  knowledge  base,  the  toolbox 
is  just  a  passive  collection  of  aids  unable  to  help  the  decision  maker 
when  he  does  not  know  how  to  proceed;  without  the  toolbox,  the  knowledge 
base  can  have  no  operational  expression. 

However,  building  of  the  environment  should  be  accompanied  by  a 
careful  experimental  learning  and  testing  program.  It  is  important  to 
evaluate  concepts  and  tools  as  they  are  added  to  the  environment,  with 
both  formal  experiments  and  the  collection  of  verbal  protocols  capable 
of  providing  valuable  data. 

In  order  to  achieve  these  goals,  seven  subtasks  need  to  be 
performed. 

(1)  Perform  research  on  theories  of  the  decision  making  process.  These 
should  have  sufficient  power  that  they  can  account  for  a  variety  of 
observed  decision  making  behaviors  and  will  allow  the  construction 
of  the  knowledge  base.  Einbedded  in  the  knowledge  base  will  be 
information  about  cognitive  limitations,  personal  decision  styles 
and  tool  preferences,  as  well  as  normative  procedures  for  decision 
making . 
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(2)  Choose  a  specific  class  of  decision  makers  and  define  experimental 
decision  situations  germane  to  that  class. 

(3)  Define  and  build  problem  structuring  aids.  Particular  attention 
should  be  paid  to  the  integrated  use  of  multiple  types  and  sources 
of  information,  and  to  the  representation  of  this  information  in 
human  readable  forms.  These  aids  need  to  build  upon  existing  work 
in  the  areas  of  decision  templates,  decision  trees,  and  influence 
diagrams. 

(4)  Enhance  the  implementation  of  AHP  paying  particular  attention  to 
the  man-machine  interface  questions. 

(5)  Define  and  build  a  variety  of  aids  for  stimulating  option 
generation  creativity.  These  aids  should  be  designed  to  facilitate 
the  capturing  of  ideas  as  they  occur,  rather  than  in  rigidly 
predefined  orders  and  formats.  Etophasis  should  be  placed  on.  but 
not  limited  to,  deriving  acceptable  computerized  versions  of  tools 
used  in  Brainstorming,  Synectics,  Conceptual  Blockbusting  and 
Lateral  Thinking. 

(6)  Define  and  build  option  evaluation  aids.  These  need  to  be  at  a 
variety  of  levels  of  detail,  from  subjective  evaluation  protocols 
to  detailed  models.  The  aids  need  to  build  on  existing  vork  in  the 
general  areas  of  value  elicitation,  and  the  specific  models 
available  in  the  domain  of  the  specific  class  of  decision  makers 
selected. 

(7)  Research  and  define  consistent  valid  measures  of  option  generation 
performance  in  complex  decision  situations. 
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7.3.  Impact  on  Air  Force  Decision  Making 


On  the  basis  of  the  insights  gained  performing  the  research  tasks 
enumerated  in  Section  7.1  above,  we  believe  the  development  and  use  of 
operational  Option  Prompting  Environments  containing  robust  knowledge 
bases,  diverse  selections  of  tools,  and  intelligent  man-machine 
interfaces  would  impact  decision  making  situations  in  the  Air  Force 
along  many  dimensions.  The  classes  of  decision  makers  affected  could 
range  from  complex  equipment  or  system  operators  to  generals  and  staffs 
making  command  decisions.  The  types  of  decision  situations  impacted 
could  range  from  simple  "closed"  decision  situations  (e.g.,  recognizing 
the  state  of  a  system  in  a  training  enviroiment  and  implementing  a 
single  "school"  action  prescribed  for  that  state)  to  complex  "open" 
decision  situations  in  an  operational  enviroment  wherein  there  are  no 
satisficing  options  evident  at  first.  The  decision  making  process  in 
each  specific  decision  situation  may  have  more  attention  focussed  on  it. 
Undesirable  cognitive  biases  could  be  procedural ly  mitigated.  A  wide 
variety  of  cognitive  styles  could  be  accommodated.  The  decision 
processes  could  more  easily  be  studied,  resulting  in  decision  theory 
extensions  and  methodology  improvements. 

Our  basic  research  was  kept  at  a  generic  level  for  the  most  part. 
However  most  of  the  specific  military  decision  situations  posed  during 
the  study  involved  command  decisions  at  the  General  level.  Such  decision 
situations  may  often  be  characterized  by  a  large  number  of  important 
decision  factors,  some  of  which  can  only  be  qualitatively  assessed, 
while  several  others  have  uncertainty  associated.  The  decision  process 
is  often  conducted  by  the  General  and  his  staff,  some  of  whom  are 
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"nuggets"  in  the  command  level  decision  processes.  We  believe  that  an 
Option  Prompting  Environment  supporting  the  General  and  his  staff  would 
be  valuable  to  both  the  experienced  member  and  the  neophyte  in  the 
organization.  The  experienced  decision  maker  would  be  aided  in  quickly 
gaining  data,  structuring  his  decision  problem,  and  in  allocating 
appropriate  time,  effort  and  staff  to  considering  the  various  decision 
factors  and  phases  of  the  decision  process.  He  would  also  benefit  from 
mechanisms  to  avoid  the  effects  of,  or  at  least  be  aware  of,  cognitive 
biases  such  as  the  anchoring,  availability,  and  selective  perception 
biases.  The  Option  Prompting  Environment  should  encourage  the  decision 
maker  to  be  creative  with  respect  to  the  many  aspects  of  difficult 
decision  situations.  It  should  help  him  check  for  the  completeness  of 
his  approach  and  thereby  provide  a  sense  of  confidence  and  motivation. 
The  inexperienced  staff  could  benefit  frcm  templates  and  knowledge  based 
guidance  which  the  Option  Prompting  Environment  would  provide.  It 
would  also  provide  a  vehicle  for  communication  and  on-the-job  training. 

In  formal  training  the  Option  Prompting  Environnent  should  be 
beneficial  in  prompting  the  decision  maker  trainee,  when  he  is  stuck,  to 
select  and  begin  a  next  appropriate  action,  instructing  him  on  the  use 
of  any  associated  tool.  It  could  also  point  out  when  the  trainee  is 
likely  to  suffer  the  effects  of  the  various  types  of  biases.  For 
example,  an  evaluation  tool  may  be  an  available  low  fidelity  model  of  an 
important  part  of  the  decision  situation.  The  decision  maker  is  likely 
to  use  the  specific  quantitative  output  of  this  model  in  his  other 
considerations  and  that  output  may  produce  an  anchoring  bias  when 
considering  even  larger  variations  on  the  decision  situation  modelled. 
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In  lover  level  decisions,  such  ss  complex  equipment  or  system 


control  decisions,  the  operator  (i.e.,  the  decision  maker)  upon 
recognizing  the  state  of  his  system  in  the  environment,  may  select  an 
option  specified  in  his  training.  Hence  the  operational  decision  making 
is  rather  straightforward.  However,  as  conditions  and/or  equipment  are 
modified,  the  researchers/developers/tacticians/trainers  that  prescribed 
the  specified  option  should  occasionally  reconsider  the  option 
prescribed.  At  that  time  the  Option  Prompting  Environment  could  be 
important,  since  the  possibly  new  option  to  be  prescribed  may 
subsequently  be  used  repeatedly  by  a  large  number  of  decision  makers. 

The  Air  Force  is  willing  to  invest  relatively  large  amounts  of 
money,  time  and  effort  in  the  explicit  training  of  pilots,  other 
operator  personnel,  and  technicians;  but  little  for  explicit  training  of 
Generals  and  staff  officers  in  one  of  their  most  important  activities: 
command  decision  making.  Providing  decision  Bupport  systems  such  as 
Option  Prompting  Environments  would  provide  mechanisms  for  both  staff 
colleges  and  operational  staffs  to  focus  efforts  in  this  area.  In 
particular,  it  may  encourage  them  to  focus  on  making  "better"  decisions 
(i.e.,  generating  and  selecting  a  "good"  alternative)  instead  of 
immediately  developing  a  plan  for  an  "OK"  alternative. 

Command  decision  making  style  is  clearly  dependent  on  the  specific 
decision  makers  involved.  If  a  decision  maker  decides  to  use  a  system, 
such  as  an  Options  Prompting  Environment,  then  he  must  submit  to  the 
constraints  inherent  in  the  system,  thereby  changing  his  style.  If  this 
change  is  substantial  it  may  affect  his  capability  to  operate  creatively 
or  efficiently.  If  he  is  willing  to  use  a  system,  then  he  should  be 
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able  to  perform  at  least  as  veil  and  as  confidently  as  without  the 
system.  Of  course  the  amount  of  training  he  has  experienced  may 
significantly  impact  his  effectiveness  with  the  system.  Flexibility 
within  the  system,  such  as  a  selection  of  different  tools  and  a  user 
friendly  interface,  may  circumvent  any  overall  stifling  effects  while 
providing  specific  useful  information. 

The  final  significant  anticipated  impact  stated  here  relates  to 
more  research  and  development  in  decision  making,  to  which  the 
generation  process  can  be  so  critical.  With  current  aircraft,  sensor, 
communication,  navigation,  and  logistic  support  systems,  command  and 
control  has  become  the  most  critical  element  of  many  Air  Force  unit 
operations;  and  perhaps  decision  making  is  the  most  important  sub¬ 
element.  A  physical  and  software  system,  such  as  an  Options  Prompting 
Enviromnent,  will  be  identified  explicitly  with  this  critical  warfare 
functional  element.  Such  an  identifiable  entity  should  focus  attention 
and  foster  efforts  to  achieve  both  deeper  understanding  of  and  better 
procedures  for  operational  Air  Force  decision  makers. 
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ANNOTATED  BIBLIOGRAPHY 


The  literature  that  is  potentially  relevant  to  Options  Generation 
is  vast,  and  it  is  clearly  infeasible  to  gather  it  all  together  in  a 
single  bibliography.  Our  goal  has  thus  been  to  construct  a  set  of 
references  that  reflect  the  major  relevant  disciplines  involved.  We  have 
included  those  publications  which  have  had  most  influence  on  our 
thinking,  several  other  key  papers  and  a  variety  of  secondary  studies. 
Also  included  are  some  review  papers  and  texts,  each  of  which  contains 
its  own  extensive  bibliography.  In  this  way,  we  provide  a  comprehensive 
introduction  to  the  literature. 


[1]  R.L.Ackoff,  E. Vergara. 

Creativity  in  Problem  Solving  and  Planning:  A  Review. 

Euro.  J.  0£.  Research  7(1). 1—13,  1981. 

This  is  a  review  article  that  discusses  some  issues  of  creativity 
in  management.  It  contains  a  brief  description  of  various 
creativity  technique;  a  secondary  reference  on  creativity. 

[2]  J .L . Adams. 

Conceptual  Blockbusting . 

Norton,  New  York,  1978. 

A  book  written  for  the  layman.  It  describes  various  procedures  for 
improving  personal  problem  solving  behavior  and  focusses  on  the 
idea  of  conceptual  blocks  which  are  similar  to  the  concept  of 
biases.  An  important  reference  on  creativity. 

[3]  T.M.Amabile. 

Effects  of  Extr ins ic  Constraint  on  Creativity. 

PhD  thesis,  Stanford  University,  1977. 

Amabile's  thesis  is  valuable  for  gaining  insight  into  the  issues  in 
any  academic  discussion  of  creativity.  However,  it  is  only  of 
historical  importance  to  our  view  of  the  Options  Generation 
problem,  since  it  is  superseded  by  [4]  and  [ 5 1 . 


[4]  T.M.Amabile. 

The  Social  Psychology  of  Creativity:  A  Consensual  Assessment 
Technique. 

J.  Personality  and  Social  Psychology  (in  press). 

An  important  paper  that  describes  a  methodology  for  assessing 
creativity.  It  contains  an  extensive  bibliography  and  is  a  valuable 
secondary  reference. 

[5]  T.M.Amabile. 

The  Social  Psychology  of  Creativity:  A  Componential 
Conceptualization . 

J..  Personality  and  Social  Psychology  (in  press). 

A  key  paper  in  the  development  of  our  ideas.  It  gives  a  careful 
exposition  of  the  definition  of  creativity  and  develops  the 
componential  model  we  use  in  Chapter  3. 

[6]  A.Arbel,  R.M.Tong. 

On  the  Generation  of  Alternatives  in  Decision  Analysis  Problems. 

J.  ObI.  Res.  Soc.  33(4) :377-387,  1982. 

This  paper  describes  our  initial  thoughts  on  the  Options  Generation 
problem.  Of  interest  mainly  for  its  decision  process  model  which  is 
somewhat  different  from  the  one  used  in  this  report. 

1 7 ]  A.Arbel,  R.M.Tong. 

Generating  Decision  Options. 

In  ORSA/TIMS  Joint  National  Meeting.  Detroit,  April,  1982. 

A  brief  report  of  the  Stanford  experiment.  Superseded  by  Chapter  6 
of  this  report. 

[8]  S. Barclay  et  al . 

Handbook  for  Decision  Analysis . 

Technical  Report  TR-77-6-30,  Decisions  and  Designs  Inc.,  1977. 

A  basic  reference  to  the  practice  of  decision  analysis  as  advocated 
by  Decisions  and  Designs  Inc.  It  contains  many  interesting  examples 
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A  good  collection  of  papers  that  cover  a  large  number  of  issues  in 
behavioral  decision  theory.  Excellent  background  material. 

[79]  S.R. Watson,  R.V. Brown. 

The  Valuation  of  Decision  Analysis. 

J.  Royal  Stat.  Soc. .  Ser.  A  141 (l) :69-78,  1978. 

A  decision  analysis  paper  that  attempts  to  formalize  a  procedure 
for  assessing  the  value  of  actually  performing  a  decision  analysis; 
marginally  relevant. 

[80]  W  .A.Wicklegren. 

Cognitive  Psychology. 

Prentice-Hall,  Englewood  Cliffs,  NJ . ,  1979. 

A  standard  text  on  the  psychology  of  problem  solving.  It  contains, 
among  other  things,  a  discussion  of  general  problem  solving 
strategies . 

[81]  R.Willensky. 

A  Model  for  Planning  in  Complex  Situations . 

Memorandum  UCB/ERL  M81/49,  Univ.  California,  Berkeley,  Electronics 
Research  Lab.,  1981. 

An  artificial  intelligence  approach  to  the  planning  problem.  This 
report  contains  some  speculations  into  the  nature  of  the  planning 
process  and  offers  a  high  level  model  that  is  to  be  implemented  in 
an  experimental  computer  program.  Only  marginally  relevant  at  this 
stage  of  our  research. 
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[82]  J  .A. Wise. 

Cognitive  Basis  for  an  Expanded  Decision  Theory. 

In  Proc .  Int .  Conf .  on  Cybernet ics  &  Society .  IEEE,  1979. 

A  discussion  of  a  model  of  human  decision  making  that  emphasises  a 
gestalt  psychology  approach.  It  argues  that  if  a  problem  is  well 
structured  then  it  is  obvious  what  the  appropriate  course  of  action 
should  be. 

[83]  H.A.Witkin 

Individual  Differences  in  Ease  of  Perception  of  Embedded  Figures. 
J_.  Personal  ity  19:1-15,  1950. 

A  classic  paper  that  provides  some  of  the  justification  for  the 
Embedded  Figures  Test  of  [84]. 

[84]  H.A.Witkin,  P.K.Oltman,  E. Raskin,  S. A. Karp 
Embedded  F igures  Test . 

Consulting  Psychologists  Press,  Palo  Alto,  CA.,  1971. 

The  basic  introduction  to  the  Embeddd  Figures  Test  used  in  our 
Stanford  Experiment. 

[85]  J.G.Wohl. 

Force  Mangement  Decision  Requirements  for  Air  Force  Tactical 
Command  -nd  Control. 

IEEE  Trans .  Systems .  Man .  and  Cybernetics 
SMC-1K19)  :618-639,  1981. 

A  key  paper.  An  excellent  discussion  of  the  need  for  decision  aids 
in  Air  Force  Tactical  Command  and  Control.  It  contains  several  good 
characterizations  of  the  issues  and  a  descriptive  model  of  the 
decision  making  process  that  helps  focus  on  the  key  problems. 

[86]  R.M.Yerkes,  J.D. Dodson 

The  Relation  of  Strength  of  Stimulus  to  Rapidity  of  Habit 
Formation 

J_.  Comparative  Neurology  of  Psychology  18:459-482,  1908. 

A  classic  psychology  paper  that  introduces  the  idea  that 
performance  is  a  curvilinear  function  of  arousal. 

[87]  R.W.Zmud. 

Individual  Differences  and  MIS  Success:  A  Review  of 
Empirical  Literature. 

Management  Science  25:966-979,  1979. 

A  valuable  review  of  the  experimental  evidence  both  for  and  against 
the  importance  of  individual  decision  style  in  determining  the 
value  of  various  management  information  systems;  good  background 
reading. 


APPENDIX  A:  SUMMARIZED  BIBLIOGRAPHY 


In  this  appendix  we  summarize  the  comments  contained  in  the 
annotated  bibliography  by  constructing  a  pair  of  cross-referenced  tables 
to  the  bibliography  entries.  Our  scheme  for  categorizing  the  literature 
is  given  in  Table  A1  where  we  show  three  major  categories  and  several 
sub-categories.  Section  2.1  of  the  main  report  gives  a  detailed 
definition  of  our  terms  and  a  discussion  of  our  reasons  for  choosing 
them.  Table  A2  summarizes  each  entry  by  placing  it  in  appropriate 
categories  from  Table  A1 ;  for  example,  entry  #1  is  a  review  of 
creativity  and  problem  solving  in  a  management  science  context,  whereas 
entry  #2  is  a  collection  of  techniques  for  improving  individual  creative 
thinking  and  problem  solving  ability.  Table  A3  lists  entries  by  topic 
and  importance  and  can  be  used  to  access  the  literature  if  a  subject  is 
known.  So,  entry  #18  is  a  primary  reference  for  cognitive  biases  and 
styles,  and  entry  #1  is  a  secondary  reference  for  creativity.  Notice 
that  the  topics  are  not  identical  to  those  used  in  Table  A2.  This  allows 
us  to  provide  a  more  flexible  breakdown  of  the  literature.  In  both  these 
tables,  bibliography  entries  may  appear  in  more  than  one  category, 
reflecting  the  wide  ranging  nature  of  some  studies. 
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tables,  bibliography  entries  may  appear  in  more  than  one  category, 
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Table  Al.  Literature  Categories 


Table  A2.  Literature  by  Category 
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Table  A3  Literature  by  Topics 


TOPIC 

PRIMARY  REFERENCES 

SECONDARY  REFERENCES 

Cognitive  biases 

19,  32,  33,  44,  45,  71, 

76,  77 

18 

Cognitive  style 

19,  32,  36,  41,  45,  71, 

87 

12,  74 

Decision  aids 

27,  39,  42,  43,  46,  53, 

58,  59,  85 

13,  30,  49,  54,  62,  66, 

72 

Group  decision  making 

15,  26,  42,  48,  57 

32,  40,  69,  70,  72 

Theories  of  decision 

making 

8,  19,  29,  31,  37,  40, 

64,  67 

17,  20,  21,  28,  34,  35, 

51,  65,  68,  70,  79,  81, 

85 

Creativi ty 

2,  4,  5,  14,  26,  47,  57 

1,  3,  9,  10,  22,  38,  50, 

63,  73,  82 

Options  generation 

6,  23,  24,  25,  52,  60, 

61 

7,  75 
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APPENDIX  B:  A  MATHEMATICAL  DESCRIPTION  OF  OPTION  GENERATION 


In  this  appendix  we  present  a  mathematical  model  of  option  genera¬ 
tion.  Its  purpose  is  mainly  illustrative,  and  we  make  no  claims  for  its 
psychological  validity.  Its  value,  however,  is  that  it  enables  us  to 
improve  our  understanding  of  the  areas  in  which  an  option  prompting 
environment  could  provide  support. 

We  start  by  considering  the  problem  recognition  phase  of  decision 
making  and  develop  a  model  that  captures  the  concept  of  "unsatisfactory 
plausible  outcomes".  This  model  naturally  leads  us  to  a  second  one  that 
describes  the  dynamic  nature  of  the  options  generation  process  in  terms 
of  an  neuristic  search  algorithm.  In  both  models,  we  see  how  the  formu¬ 
lation  can  be  used  to  define  appropriate  decision  aids. 

Problem  Recognition 

In  any  situation,  the  first  question  that  a  decision  maker  faces  is 
"Do  I  have  a  decision  to  make?"  If  he  is  being  instructed  to  deal  with 
a  specific  problem  in  a  specific  way  then  the  answer  to  this  is  trivial. 
However,  in  many  situations  the  answer  is  not.  We  believe  that  in  many 
cases  the  ability  to  generate  good  options  hinges  on  early  detection  of 
the  decision  problem. 

We  need  to  distinguish  between  two  kinds  of  activity.  In  the  first, 
or  "tactical",  activity  simple  adjustments  are  made  to  a  nominal  course 
of  action  in  order  to  adapt  to  a  real-time  situation.  In  the  other,  or 
"strategic",  activity  a  more  involved  thinking  process  is  carried  out  to 
determine  a  new  course  of  action.  We  shall  refer  to  the  former  activity 
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as  an  operational  procedure  and  the  latter  aa  option  generation.  Usu¬ 


ally  an  operational  procedure  requires  only  local  and  micro  information 
about  the  deviation  of  the  actual  activity  from  the  nominal  course  of 
action.  Options  Generation,  on  the  other  hand,  requires  global  but  macro 
information  about  the  overall  situation. 

Let  us  consider  a  decision  maker  at  a  specific  level  in  the  deci¬ 
sion  hierarchy.  We  assume  that  a  certain  nominal  course  of  action  is 
being  caken  and  that  the  operational  procedures  are  carried  out  in 
real-time.  The  question  now  is,  when  will  the  decision  maker  think  chat 
the  nominal  course  of  action  is  inappropriate  and  that  a  new  course  of 
action  needs  to  be  determined.  To  answer  this,  we  shall  extend  Simon's 
satisficing  model  and  relate  it  to  the  option  generation  process.  We 
argue  that  due  to  bounded  rationality,  the  decision  maker  tendB  to  con¬ 
tinue  with  an  established  operational  procedure  so  long  as  he  is  satis¬ 
fied  with  the  plausible  outcomes  resulting  from  that  procedure.  So,  the 
issues  are:  what  are  plausible  outcomes,  and  when  is  the  decision  maker 
dissatisfied  with  them. 

Let  us  first  discuss  the  notion  of  a  plausible  outcome.  Assume  that 
a  nominal  course  of  action  is  to  be  taken.  However,  due  to  the  fact  that 
the  future  event  is  uncertain,  or  is  being  manipulated  by  the  other 
players  in  the  environment,  an  operational  procedure  has  the  character 
of  a  simple  feedback  mechanism.  As  a  result,  the  decision  maker  derives 
a  set  of  plausible  outcomes.  To  represent  this  symbolically,  let  us 
denote  the  future  event  by  w,  and  the  adjustment  feedback  mechanism  by 
u(.).  Then  the  outcome  caused  by  w  and  u(w)  is  denoted  by  y(u(w),w).  If 
the  decision  maker  anticipates  that  the  future  event  will  be  within  the 
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sec  ft  ,  Chen  Che  sec  of  plausible  outcomes  will  be 


Y  *  {y(u(w),  w):weft} 


The  desirability  of  a  cerCain  outcome  may  be  represented  by  a  set 
of  attributes,  and  we  let  a-{aj , . . . ,an>  be  the  list  of  attributes  that 
the  decision  maker  is  concerned  with.  We  may  now  imagine  a  mapping 
a(y )={a^ (y afl(y) }  which  represents  the  desirability  of  the  outcome 

y.  Thus  with  the  set  of  plausible  outcomes  Y,  we  can  associate  with  it  a 

A 

subset  a(Y)={a(y ) :y e  Y}  in  the  attribute  space. 

Now  we  are  in  a  position  to  discuss  the  notion  of  satisfaction.  It 

is  difficult  to  define  satisfaction  in  terms  of  a  measurable  quantity, 

so  we  will  define  it  in  terms  of  an  aspiration  level,  a  ={a. . a  >,  in 

In 

the  attribute  space.  This  is  a  subjective  notion  which  represents  the 

ic 

level  that  the  decision  maker  is  eager  to  achieve.  In  many  case,  a  is 

specified,  or  implied  by  the  decision  maker's  superior.  In  other  cases, 

it  is  derived  from  such  factors  as  past  achievement  records,  his 

competitor's  performance  records,  and  other's  expectation  of  him.  For 

the  present  discussion,  we  shall  not  concern  ourselves  with  how  the 

* 

decision  maker  derives  his  a  ;  rather,  we  shall  discuss  the  consequences 

★ 

of  naving  a  particular  a  . 


Let  us  concentrate  on  the  attribute  space.  For  illustrative  pur¬ 
pose,  let  us  consider  the  situation  where  there  are  two  attributes 
(8^,82)  and  the  subset  a(Y)  is  given  by  A  in  Figure  Bl.  We  assume  than 

an  increase  in  the  value  of  each  attribute  will  increase  satisfaction. 

* 

Referring  to  Figure  Bl,  if  the  aspiration  level  a  is  located  at  P. , 
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is  located  at  P^, 


Fig.  Bl.  Attribute  space 


then  we  see  that  for  all  plausible  outcomes  Y,  the  associated  attributes 

exceed  the  aspiration  level,  and  thus  the  decision  maker  would  be  satis- 

* 

f ied  with  the  current  operational  procedure.  If,  however,  a  is  located 
at  '?2»  then  the  decision  maker  would  see  that  under  no  circumstances 
will  the  current  operational  procedure  allow  him  to  achieve  his  aspira¬ 
tion  level.  This  causes  dissatisfaction  with  the  current  operational 
_  * 

procedure.  The  ambiguous  situation  is  when  a  is  at  P^.  Then,  depending 
upon  what  plausible  outcome  is  actually  realized,  the  decision  maker  may 
or  may  not  be  satisfied.  Therefore,  whether  the  present  operational 
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procedure  is  satisfactory,  or  not,  hinges  on  whether  the  decision  maker 
can  better  confine  the  set  of  plausible  outcomes.  That  is,  reduce  Y 
which  in  turn  shrinks  a(Y). 

This  discussion  shows  that  dissatisfaction  is  not  necessarily  a 
result  ot  ambiguity  in  plausible  outcomes,  but  rather  the  possibility 
that  certain  plausible  outcomes  fall  short  of  the  aspiration  level.  Let 
us  detine  a  proxy  for  dissatisfaction  via  the  weighted  expectation  gap 


A  - 

n 

£  £ 

y  Y  i-1 

XiAi(a*-a(y))c(y) 

where 

Ai(x)  = 

(  fi(x) 

\  0 

if  x  >  0  ; 

x  <  0 

f^(x)  is  an  increasing 
function 

X^  «  relative  importance  of  attribute  i  (normalized  to  1) 
c(y)  *  confidence  for  outcome  y  being  true  (normalized  to  1) 

The  tunctions  {f  ^  ( • )  and  weightings  (X^K^  are  all  subjective 

and  assumed  to  remain  the  same  for  a  specific  decision  maker.  Note  that 

in  Figure  B1  if  a  is  at  P A  =  0;  whereas  A  >0  if  a  is  at  or  P^. 

“  *  ~  ★ 

Note,  however,  that  A  (a  *P^)<A(a  *Pj ) . 

Dissatisfaction  is  a  threshold  phenomenon  and  we  postulate  that  the 
decision  maker  is  unsatisfied  if 

A  >  £  >  0 


where  e  is  a  threshold  parameter.  Thus  dissatisfaction  is  a  function  of 


Che  aspiration  level  and  che  set  of  plausible  outcomes.  Next  ve  shall 
discuss  the  decision  behavior  of  the  decision  maker  when  he  is  dissatis¬ 
fied. 

Since  Y  results  from  both  ft ,  the  set  of  random  future  events,  and 
u(.),  the  operational  adjustment  rule,  then  A  can  be  expressed  as 
—  n 

A(a*.  u(-),  £(•)  -  22  g  Vi(ai  “  a(yC“(w),w)))£(„) 

where  c(w)  is  the  confidence  in  w  being  true,  and  c(w)*c(y(u(w),w) . 
Given  a  nominal  course  of  action  to  be  taken,  we  see  that  the  expecta¬ 
tion  gap  is  a  function  of 

-  the  aspiration  level, 

-  the  relative  confidence  on  the  possibility  of  different 

-  future  events,  and 

-  the  operational  adjustment  rule. 

We  postulate  that  if  A>  £,  then  the  decision  maker  will  first  attempt 
to  reduce  A  by 

-  improving  the  operational  adjustment  rule  and/or, 

-  actively  collecting  more  information  in  order  to 

reduce  the  uncertainty  about  possible  future  events. 

If  this  still  yields  a  A  >e,  then  the  decision  maker  should  realize 
that  there  is  a  problem  and  begin  the  options  generation  process.  This 
is  a  reasonable  model  if  we  assume  bounded  rationality  and  that  the  cost 
of  the  options  generation  process  is  higher  than  that  of  making  opera¬ 
tional  adjustments  and  collecting  information. 

From  this  discussion  we  see  that  non-recognition  of  the  problem  at 
the  right  time  might  be  due  to: 


-  bias  on  future  event  assesanen t, 

-  incorrect  outcome  modeling, 

-  inappropriate  aspiration  level,  and 

-  bias  on  relative  importance  of  each  attribute. 

Each  of  these  factors  might  be  important  and  in  devising  an  appropriate 
aid  that  will  facilitate  problem  recognition  we  should  consider  ways  for 
it  to : 

-  help  in  assessing  future  events, 

-  help  in  determining  possible  outcomes, 

-  display  the  attributes  associated  with  each  possible  outcome, 

-  help  to  improve  the  operational  procedure  if  needed, 

-  help  facilitate  access  to  information, 

-  help  to  set  appropriate  aspiration  levels,  and 

-  prompt  the  decision  maker  to  enter  the  options  generation 

phase  at  the  correct  time. 


The  Dynamics  of  Option  Generation 

Let  us  use  a  simple  example  to  lead  into  a  discussion  of  a  model  of 
the  option  generation  process.  Consider  a  production  problem  where  the 
goal  is  to  achieve  high  productivity.  For  simplicity,  let  us  assume  that 
a  production  process  is  represented  by 


y  -  c(x) 

where  y  is  the  output  unit,  x  is  the  activity  vector  which  contributes 
to  output  production.  It  is  also  assumed  that  if  a  certain  activity  vec¬ 
tor  x  is  selected,  the  total  resource  required  to  give  such  an  activity 

r  “  f(x)  (2) 
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Now,  Che  goal  of  high  productivity  can  have  two  interpretations. 
First,  for  a  given  resource  unit  F,  find  an  activity  vector  x*(r)  such 


y*(r)  £  c(x*(?))  >  c(x) 


V  x  s.t.  f(x)  <  r 


Second,  for  a  specific  output  level  y,  find  an  activity  vector  x°(y) 
such  that 


r°0  a  f(x°(y)  <  f(x) 


V  x  s.t.  c(x)  <_  y 


The  first  interpretation  is  to  maximize  the  output  subject  to  resource 
constraint  (the  economic  view);  whereas  the  second  is  to  minimize  the 
resource  required  to  ensure  an  output  level  (the  conservation  view). 

Mathematically  the  two  are  equivalent  if  for  a  given  r  in  the  first 

.  — A  *  _ 

interpretation,  y=y  (r)  is  used  in  the  second  interpretation.  However, 
in  general,  the  two  interpretations  will  yield  different  answers  since 
neither  the  desired  output  nor  the  resource  units  are  specified. 

In  fact,  the  goal  does  not  express  any  preference  on  the  two  attri¬ 
butes:  output  level,  resource  requirement.  In  most  cases,  the  individual 
tends  to  interpret  the  goal  statement  as  one  given  by  (1)  or  by  (2) 
depending  on  his  subjective  view  of  what  seems  to  be  the  most  important 
attribute.  Thus  a  certain  degree  of  subjectivity  is  present  in  any  deci¬ 
sion  problem.  Now,  we  can  define  the  notion  of  an  option  and  a  good 
option  analytically.  An  activity  x  is  called  an  option  for  a  given  pair 
(y,r)  if 
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The  set  of  all  options  for  (y,r)  is  denoted  by  Q  (y,r).  We  shall  further 
define  partial  ordering  on  the  set  of  activities  {x}  as  follows: 

-1  >:>  -2  if  1  c(x2)  and  f(Xj.)  £  f(x2) 

That  is,  Xj  "is  at  least  as  good  as"  x2  if  x^  uses  less  (or  equal  amount 
of)  resource  while  producing  higher  (or  equal)  output.  Also,  if  x^>>x2 
and  x2>>Xj,  we  say  they  are  equivalent,  and  are  denoted  by  x^%x2.  An 
option  x  for  (y,r)  is  said  to  be  a  good  option  for  (y,r)  if  for  all 
x  £  Q  (y,r) 

x  >>  X*  X*  =  X 

The  set  of  good  options  for  (y,r)  will  be  denoted  0  (y,r).  Given  an 

acceptable  performance  (y,r),  the  problem  of  finding  all  options 

corresponding  to  (y,r)  is  to  determine  the  set  0  (y,r);  whereas  the 

problem  of  rinding  all  good  options  corresponding  to  (y,r)  is  to  deter- 
★  —  — 

mine  the  set  ft  (y,r). 

Proposition : 

Let  x°  be  a  (good)  option  for  (y,r),  then  it  is  also  a  (good) 
option  tor  (y°,r°)  if 

y°  _<  y  ;  r°  >  r  (6) 

Proof : 

If  x°  is  an  option  for  (y,r),  then  from  (5)  and  (6)  we  have 


B9 


c(x°)  £  y  £  y°  5  f(x°)  £  r  £  r° 

and  thus  x°  is  an  option  for  (y°,r°).  This  also  implies 

n(y,  r)  c  n(y°.  r°  ) 


If  x°  is  a  good  option  for  (y,r)  then 

c(x°)  £  y  ;  f(x°)  £  r  (7) 

and  for  all  x  e  Q  <y,r) 

X  a  X  4-  o*  «  * 

o 

Let  x  $  fi  (y,r)  then  either 

c(x)  <  y  (8) 


or 


f(x)  >  r  (9) 

or  both.  Clearly  if  either  (8)  or  (9)  ior  both)  is  satisfied,  then 
x/>x°.  Therefore,  none  of  the  options  x  e  fi  (y°,r°),  x  $  ft  (y,r) 
will  be  preferred  over  x°.  Thus  if  there  exists  x  £  (y°,r°)  such 

that 

*  »  £o 


BIO 


r 


then  x  must  be  in  ft  (y,r),  and  thus  x  %  xq.  This  implies  that  x°  is 

a  good  option  for  ft  (y°,r°),  and  that 

ft*  (y,  r)  9  (yo,  r°) 

Thus  the  (good)  option  set  increases  as  y  decreases  and  r 
increases.  Intuitively,  this  means  that  if  we  are  demanding  less  output 
production  while  increasing  the  amount  of  usable  resources,  the  set  of 
(good)  options  increases.  Conversely,  if  we  are  demanding  high  output 
production  while  reducing  the  amount  of  usable  resources,  the  set  of 
(good)  options  decreases.  It  is  conceivable  that  when  the  output  demand 
is  coo  high  while  the  usable  resource  is  too  low,  we  have  no  option. 

Let  (y,r)  be  a  specified  pair  such  that  ft  (y,r)"0,  then  we  shall 
say  that  such  specification  is  inconsistent  with  the  productive  capabil¬ 
ity.  All  the  above  concepts  can  be  illustrated  via  Figure  B2. 

For  a  given  pair  (y,r),  if  ft  (y,r)^0,  then  it  is  relatively 
straightforward  to  determine  ft  (y,r)  and  ft  (y,r)  if  the  functions  c(.) 
and  f(.)  are  given  analytically.  However,  the  main  problem  in  practice 
is  that  neither  function  is  given,  and  so  if  x  is  specified,  their 
evaluation  is  a  time  consuming  exercise.  Under  such  a  situation,  a 
search  procedure  must  be  adopted  in  finding  ft  (y,r)  and  ft  *(y,r). 

There  are  many  search  algorithms  that  will  allow  one  to  find  points 
_  _  *  _ 

in  ft  (y,r)  and  ft  (y,r),  but  we  will  concentrate  on  those  which  are 


closest  to  human  search  processes.  It  is  argued  that  a  gradient  pro¬ 
cedure,  which  requires  that  both  constraints  in  (5)  are  treated  equally, 
is  not  usually  used  by  a  human  in  his  search  process  because  of  the 


limited  capability  in  human  evaluation  and  assessment.  Simon  refers  to 
this  as  limited  rationality.  Belov  we  describe  a  model  for  options  gen¬ 
eration  that  bears  some  resemblance  to  that  proposed  by  Simon. 

First,  it  is  assumed  that  the  decision  maker  will  single  out  one  of 
the  constraints  as  an  attribute  for  attention.  He  then  determines  an 
aspiration  level  for  this  attribute.  In  our  example,  let  us  assume  that 
he  singles  out  production  as  an  attribute  and  sets  an  aspiration  level 

output  2.  y 


Next,  he  would  list  all  the  activities  that  influence  the  level  of  out¬ 


put,  denoting  such  activities  by  {x^,...,xq}.  Then  he  would  determine  an 
appropriate  combination  of  the  activities  such  that 

c(x)  >.  y 

If  there  are  no  feasible  constraints  on  each  activity  this  i6  usually 
possible.  Otherwise  he  may  find  that  no  such  combination  exists,  in 
which  case  he  will  reduce  the  aspiration  y.  The  process  iterates  until 
(10)  is  satisfied. 

Next,  he  would  determine  the  amount  of  resource  units  that  he  would 
not  want  to  exceed: 

resource  used  <_  r 

If  the  x  which  is  satisfied  by  the  above  process  is  such  that 

f(x)  <  r  (11) 

then  he  ha6  found  a  point  in  (1  (y,r).  If,  however,  (11)  is  violated, 
then  he  would  focus  on  the  resource  requirement  and  try  to  generate  a 
new  combination  x'  such  that 

f(x')  <  r 

and  then  test  to  see  whether  x'  satisfies  the  output  requirement.  The 
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choice  of  x'  is  usually  obtained  by  perturbing  the  components  of  x  one 
at  a  tine.  If  the  functions  c(.)  and  f(.)  are  reasonably  well  behaved 
(convex  and  concave  respectively),  then  the  above  iterative  process  will 
lead  to  an  option  in  0  (y,r)  (region  A  in  Figure  B2)  if  the  option  set 
is  non-empty.  Otherwise,  the  decision  maker  has  to  increase  r. 

The  decision  maker  can,  instead  of  focusing  an  production,  select 
the  resource  constraint  on  the  first  pass  and  then  iterate  as  above. 
This  may  lead  to  a  different  set  of  options  (region  B  in  Figure  B2)  and 
good  options. 

If  we  believe  that  the  options  generation  process  can  be 
represented  by  the  iterative  procedure  just  described,  then  we  can  dis¬ 
cuss  the  issues  underlying  poor  option  generation  performance. 

First,  the  decision  maker  may  not  have  a  complete  list  of  attri¬ 
butes.  This  is  usually  due  to  the  fact  that  that  each  individual  nas  a 
subjective  view  of  what  the  decision  problem  is  and  which  attributes  are 
important.  If  the  list  is  incomplete  then  during  the  iterative  process, 
all  the  options  generated  may  fall  short  in  an  attribute  that  is 
neglected  in  the  whole  analysis  but  which  is  extremely  relevant  to  the 
decision  problem. 

Second,  there  may  be  a  bias  on  the  initial  choice  of  attribute. 
The  first  to  be  singled  out  represents  the  decision  maker's  subjective 
view  or  which  attribute  is  most  important  for  the  decision  problem. 
Sometimes  emotional  and  other  irrational  elements  are  involved  in  choos¬ 
ing  this  attribute,  and  if  the  determination  of  aspiration  level  for 
this  attribute  is  too  high,  then  this  may  result  in  a  few  options  which 
completely  neglect  the  other  attributes. 
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Third,  there  may  be  incomplete  determination  of  all  the  activities 
which  will  contribute  to  the  particular  attribute  under  consideration. 
This  may  be  due  to  lack  of  knowledge. 

Fourth,  there  may  be  erroneous  determination  determination  of  the 
effect  of  the  chosen  combination  of  activities  to  different  attributes. 
This  is  due  to  a  wrong  model  of  the  real  situation.  This  would  typically 
be  the  case  if  the  outcome  is  due  to  collective  actions  by  different 
decision  makers. 

Then  finally,  the  decision  maker  may  have  an  inefficient  search 
procedure.  This  i6  due  to  the  limited  capability  of  an  individual  to 
carry  out  a  search  by  varying  several  activity  components  all  at  once. 


APPENDIX  C:  THE  ANALYTIC  HIERARCHY  PROCESS 


The  hierarchical  value  assessment  scheme  presented  here  is  based  on 
Saaty's  approach  to  hierarchical  decision  problems[65] •  We  shall  review 
just  the  basics  of  the  approach. 

In  the  Analytic  Hierarchy  Process  (AHP)  a  decision  problem  is 
decomposed  into  levels  containing  objects  with  similar  attributes.  (A 
level  describing  objectives  and  a  level  describing  policies  designed  to 
meet  these  objectives,  for  example.)  The  approach  is  to  assign  local 
priorities  for  each  member  of  a  particular  level.  In  particular,  one  is 
interested  in  ways  to  propagate  the  local  priorities  of  each  level 
throughout  the  hierarchy  to  establish  global  priorities  in  a  particular 
level  of  interest  which  are  a  measure  of  the  value  of  the  attribute  to 
the  achievement  of  the  overall  goal . 

Consider,  for  example,  the  situation  described  in  Figure  Cl.  The 
decision  maker  has  to  evaluate  the  relative  importance  (priorities)  of 
the  three  policies  under  consideration;  this  in  turn  will  help  him  later 
in  allocating  resources  to  implement  these  policies.  The  policies  them¬ 
selves  are  designed  to  meet  certain  objectives  (two  in  this  particular 
case)  that  contribute  to  the  overall  goal  the  decision  maker  is  trying 
to  attain. 

In  constructing  hierarchical  structures  other  than  the  one  shown  in 
Figure  Cl,  the  following  guidelines  should  be  remembered: 

(1)  The  number  of  levels  used  in  a  particular  hierarchy  is  not  fixed 

and  should  be  chosen  to  reflect  the  particular  problem  at  band. 


Cl 
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Fig.  Cl  Hierarchical  Policy  Evaluation 


(2)  The  order  of  the  levels  should  be  one  that  reflects  a  logical 
causal  relationship  between  adjacent  levels. 

(31  The  number  of  members  in  a  particular  level  should  be  chosen  to 
describe  the  level  in  adequate  detail. 

The  points  mentioned  above  indicate  that  the  construction  of  a  par¬ 
ticular  hierarchy  is  not  a  process  that  follows  rigid  rules,  but  rather 
adapts  itself  to  the  situation  at  hand. 
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Deriving  the  actual  local  priorities  of  members  in  each  level  is 
done  through  a  pairwise  between  each  member  of  the  level  relative  to  a 
member  of  the  adjacent  upper  level. 

Basic  Procedures 

Let  us  start  the  technical  discussion  by  demonstrating  the  deriva¬ 
tion  ot  the  priorities  among  a  set  of  activities.  For  illustrative  pur¬ 
poses,  let  us  consider  three  activities  denoted  by  A^,  i-1,2,3.  We  will 
compare  the  contribution  of  these  activities  to  a  certain  objective. 
This  comparison  will  be  carried  out  pairwise  and  the  result  of  the  com¬ 
parison  will  yield  the  relative  weight,  w^,  of  the  activities  under  con¬ 
sideration.  This  pairwise  comparison  can  be  summarized  in  a  comparison 
matrix  A  given  by 


"l/Vj 

Wl/v2 

Wl/v3 

A  = 

W2/v'1 

W2/v2 

W2/w3 

1 

W3/Wl 

V3/v2 

W3/w3 

The  information  displayed 

in  this 

matrix  is 

every  element,  ay>  shows  the  relative  contribution  to  the  objective  of 
the  i-th  activity  compared  to  the  j-th  activity.  That  is 

a .  .  A  Wi  1  <  i<n  ,  l <  i<n , 

,J'=J  "  " 
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This  definition  indicates  that 


1 


which  results  in  the  matrix  A  being  a  reciprocal  matrix.  Note  also  that 
the  diagonal  elements  of  matrix  A  are  all  l's. 

Returning  to  the  example  of  Figure  Cl,  we  can  construct  a  com¬ 
parison  matrix  that  shows  how  each  of  the  three  policies  contribute  to 
objective  #1,  say.  Every  element  of  this  matrix  can  be  obtained  using  a 
question  of  the  form 

"Consider  policy  #1  and  policy  #2,  which  one  contributes  more 
towards  objective  #1,  and  what  is  the  strength  of  this  contribu¬ 
tion?" 

Whenever  the  ij-th  element  of  the  matrix  is  entered,  the  ji-th  position 
is  automatically  entered  with  the  reciprocal  value. 

To  recover  the  weights,  w^,  rather  than  their  ratios  we  proceed  as 
follows.  Note  that 


Aw  =  nw 


and  since  we  can  write  A  as  the  outer  product  of  two  vectors 


w 


w 


1 

2 


A  * 


[l/w1  l/w2  ...  l/wn]  A  Aj  A2 


then  the  eigenvalue*  of  are  given  by 


n<A>  A  I  A -XIn  I  -  I  AxA2  -  Xln  I  -  (Xn_1)|  A2ArX|  =  An_1(n-X)  -  0 

Hence  a  comparison  matrix  has  (n-1)  of  its  eigenvalues  at  the  origin  and 
the  n-th  eigenvalue  is  equal  to  the  dimension  of  the  matrix. 

since  X  “n  is  the  largest  eigenvalue,  we  conclude  that  the  vector 
of  priorities,  w,  is  simply  the  largest  eigenvector  of  the  matrix  A 
corresponding  to  the  largest  eigenvalue.  Since  we  are  interested  in  a 
relative  ordering,  this  eigenvector  is  normalized  so  that  its  components 
sum  to  1.0. 

There  are  three  questions  that  need  to  be  asked  at  this  point: 

(11  How  does  the  decision  maker  quantify  his  judgement  of  the  "strength 
of  contribution"  of  a  certain  activity? 

(2)  How  do  we  define  consistency  in  this  judgement  elicitation  pro¬ 
cess? 

(3)  How  do  we  proceed  with  the  process  across  and  beyond  a  given  level? 

These  questions  will  be  discussed  briefly  in  the  ranainder  of  this 
appendix. 

The  comparison  process  elicits  qualitative  judgemental  statements 
that  indicate  the  strength  ot  the  decision  maker's  preference  in  the 
particular  comparison  made.  In  order  to  translate  these  qualitative 
statements  into  numbers  to  be  manipulated  to  establish  the  required 


priorities,  a  reliable  scale  has  to  be  established.  Much  work  has  been 
done  on  the  subject  of  scales  in  preference  statements  and  we  will  not 
repeat  the  arguments  that  lead  to  the  employment  of  a  particular  scale. 
Instead  we  will  present  a  scale  that  is  useful  within  the  AH P  paradigm. 
This  is  shown  in  Table  Cl. 

When  using  this  scale,  we  replace  a  qualitative  comparison  state¬ 
ment  with  the  appropriate  quantifier.  For  example,  if  policy  #1  is 
weakly  preferred  to  policy  #2  then  a12“3  (and  by  the  reciprocity  pro¬ 
perty  of  A,  a2j*l/3).  Performing  the  complete  pairwise  comparison  for 
all  three  policies  will  result  in  a  3-by-3  matrix  the  normalized  eigen¬ 
vector  ot  which  yield  the  importance  of  the  three  policies. 

In  comparing  activities,  we  expect  that  if  activity  A^  is  preferred 
to  activity  A2,  and  a2  is  preferred  to  activity  A^,  then  activity  Aj 
should  be  preferred  to  activity  A^.  In  employing  a  numerical  scale,  this 
transitivity  should  be  maintained  throughout  the  comparison  process. 
Mathematically  we  define  this  as 


ik  kj 


Vi.j.k  c  (1,2,... ,n } 


This  is  simple  to  understand  when  we  recall  that 


So,  if  we  have  already  established  the  relative  strength  of  the  i-th 
activity  compared  to  the  k-th,  and  the  k-th  compared  to  the  j-th,  then 
this  should  also  yield  the  comparison  of  the  i-th  to  the  j-th.  That  is 


I-Vjt, 


Table  Cl  Comparison  Scale 


Intensity  of 

Importance 

Definition 

Explanation 

1 

Equal  importance 

Two  activities  contribute 
equally  to  the  objective 

3 

Weak  importance  of 
one  over  another 

Experience  and  judgement 
slightly  favor  one  activity 
over  another 

5 

Essential  or  strong 
importance 

Experience  and  judgement 
strongly  favor  one  activity 
over  another 

7 

Very  strong  or  demon¬ 
strated  importance 

An  activity  is  favored  very 
strongly  over  another;  its 
dominance  demonstrated  in 
practice 

9 

Absolute  importance 

The  evidence  favoring  one 
activity  over  another  is  of 
the  highest  possible  order 
of  affirmation 

2, 4, 6, 8 

Intermediate  values 
between  adjacent  scale 
values 

When  compromise  is  needed 

Reciprocals  of  above 
nonzero 

If  activity  i  has  one 
of  the  above  nonzero 
numbers  assigned  to  it 
when  compared  with  act¬ 
ivity  j ,  then  j  has 
the  reciprocal  value 
when  compared  with  i 

A  reasonable  assumption 

a.,  a.  .  A  i  k  .  i  A  a.  . 
lk  kj  -  —  •  —  —  =  ij 
w  w  w 

j  j  j 

When  the  matrix  A  ia  consistent,  its  largest  eigenvalue  is  equal  to 
its  dimension.  But  when  it  is  not  consistent  (ie.  the  consistency  condi¬ 
tion  does  not  hold  tor  some  elements)  then  the  largest  eigenvalue  of  A 
is  always  greater  than  the  dimension. 

A  >  n 

max 


Then  the  priority  vector  is  obtained  by  solving  the  following 
eigenvector  problem  for  w 

Aw  »  X  w 
max 

A  consistency  index  is  then 


where  in  the  consistent  case  C1“0. 

Next  we  consider  how  to  propagate  priorities  through  the  hierarchy. 
Let  1  (i»0, 1 , 2, . . . ,N)  be  an  index  over  the  levels  of  the  hierarchy.  Then 
i*0  corresponds  to  the  apex  of  the  hierarchy  and  i-N  corresponds  to  the 
lowest  level.  Now  let  A.(i)  be  the  j-th  activity  (or  attribute)  in  the 
i-th  level,  Pj(i)  be  the  priority  associated  with  the  j-th  activity 
(attribute)  in  the  i-th  level  with  respect  to  the  overall  goal  (Global 
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Priority),  and  Wj^(i)  be  the  relative  strength  of  the  j-th  activity 
(attribute)  in  the  i-th  level  when  compared  with  other  activities 
(attributes)  in  the  same  level  relative  to  the  the  k-th  activity  (attri¬ 
bute)  in  the  (i-l)-th  level  (Local  Priority).  This  is  illustrated  in 


Fig.  C2  Adjacent  Levels  in  a  Hierarchy 


Figure  C2  where  we  depict  two  adjacent  levels  of  a  hierarchy. 

In  this  situation  the  priorities  in  the  (i-l)-th  level  have  been 
establisned,  and  the  next  step  is  to  find  the  priorities  associated  with 
the  activities  of  the  i-th  level.  At  every  level,  the  priorities  are 
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normalized  such  Chat 


L  V‘>  ’  1 

j 


The  relative  strength, 


v.  . , 


of  the  activities  in  the  i-th  level  are 


found  by  constructing  a  comparison  matrix  of  the  form 


Thus  Aj  is  constructed  after  a  pairwise  comparison  of  the  activities 
listed  with  respect  to  the  activity  Aj(i-l)  which  id  the  j-th  activity 
in  the  adjacent  (upper)  level.  Once  is  obtained,  we  can  solve 


A. 

J 


X  w. 
max  J 


and  the  eigenvector  Vj  is  then  normalized. 

We  should  remark  at  this  point  that: 

(1)  The  number  of  Aj  matrices  to  be  constructed  is  equal  to  the  number 
of  activities  in  the  (i-l)-th  level,  the  dimension  of  these 
matrices  is  equal  to  the  number  of  activities  in  the  i-th  level. 

(2)  The  arrows  shown  in  Figure  C2  are  drawn  to  illustrate  he  direction 
of  the  comparison. 
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Once  all  Aj  matrices  are  constructed  and  their  normalized  eigenvec¬ 
tors  are  found,  the  priorities  of  the  i-th  level  are  found  from 

P(i)  *  fw1'1w2'l  ...vj  P(i-l)  =  W(i)  P(i-l),  i-1,2 . n 

I  * 

where  W(i)  is  the  matrix  whose  columns  are  the  eigenvectors  <w^>  and  the 
priority  of  the  apex  is  such  that  P(0)*1.0. 


Alternative  Method  for  Computing  Priorities 


In  constructing  a  pairwise  comparison  matrix,  every  element,  a 


is  given  by 


ij’ 


a  .  . 


which  is  a  correct  relation  if  the  priorities  tw^}  were  known.  When  the 
elements  of  the  matrix  are  constructed  based  on  judgement,  and  the  use  of  a 
ratio  scale,  one  can  expect  slight  deviations.  In  these  cases  the  elements  of 
the  matrix  will  be  given  by 


1 1 


w . 
_ 1 

w  . 

J 


1J 


The  values  of 


iJ 


will  be  positive  and  will  not  deviate  much  from  unity. 


If  we  assume  that  the  £..  can  be  viewed  as  a  random  variable,  whose  natural 
logarithm  is  normally  distributed  around  zero,  we  can  derive  the  priorities 
as  a  normalized  geometric  mean  of  the  rows  of  the  comparison  matrix. 


Let  the  comparison  matrix  A  be  described  by 


A  = 


*-  n 


where  a^ e  1R  is  a  row  vector 


The  geometric  mean  of  a  row,  g^,  is  given  by 


gi  *  (ail  •  ai2  •**  ain) 


1/n 


n  a 

j  =  i 


1/n 


ij 
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Taking  natural  logarithm  on  both  sides 


x  =  In  g  =  ^  [ ln(a  )  +  ln(a  )  +  ■••  +  ln(a  )] 
1  in  11  lz  in 


j=i 


Since  the  priority  w^,  is  defined  ns  the  geometric  mean,  we  have 


x .  =  -  ln(a . . )  =  ln(w .  )  i=l ,2 , • • • ,n 

i  n  f-?,  lj  i 


j  =  l 


from  which  we  find 


w.  =  e 

l 


To  get  a  normalized  vector  of  priorities  we  modify  the  above  expression 
to  yield 


x . 
l 


w . 


n  x . 

£e  1 

i=l 


i=l,2, • • • ,n 


The  consistency  of  the  pairwise  comparisons  can  be  checked  through  the 
multiple  correlation  coefficient  given  by 


n  L  xj 

2  i=l 

R  = 


£  <ln  aij) 


i<j 


Results  are  acceptable  for  values  of  this  coefficient  that  are  greater 
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than  0.9. 


r 


The  largest  eigenvalue  of  the  comparison  matrix  can  be  approximated  by 

A  O2 

X  =  n  +  (n-1)  [exp  )  -1] 


where 


c  = 


XUna  )  ~n  L  0  2 

i<3 _ _ _ i~l  i 

n(n-l) 


The  advantages  of  the  logarithmic  regression  approach  are  in  the  closed 
form  expressions  for  computing  the  priority  vector  (rather  than  an  iterative 
process),  and  in  explaining  the  way  this  priority  vector  is  derived  (a  geo¬ 
metric  mean  of  a  row  vs.  an  eigenvector  corresponding  to  the  largest  eigenvalue), 

Example :  Consider  a  pairwise  comparison  matrix  given  by 


A  = 


1  2  3  2 

1/2121 
1/3  1/2  1  1/2 

1/2121 


x1  *  k  [ln(l)  +  In (2)  +  In (3)  +  ln(2)]  =  0.6212  ;  =  e  1  =  1.8612 


x2  =  ^[ln^)  +  ln(l)  +  In  (2)  +  ln(l)]  =  0.00 


$2  =  e  =1.00 


x3  =  1  [ ln(l/3)  +  InC-s)  +  ln(l)  +  ln(4)]  =  -0.6212  ;  w3  =  e  J  =  0.5373 


x4  -  k[ln(H)  +  ln( 1)  +  In (1)  +  ln(2)]  =  0.00 


The  normalized  priority  vector  is  given  by 


v1  =  [0.4231,  0.2274,  0.1222,  0.2274] 


W4  =  6 


1.00 


J 


C14 


Using  the  eigenvector  approach  (the  HPA  program)  yields 


wT  -  (0.42,  0.23,  0.12,  0.23] 


The  multiple  correlation  coefficient  is  given  by 


-  &  (*i)2 

L (ln  a  )‘ 
i<j  3 


4[ (0.6212)2  +  (0.00)2  +  (0.6212)2  +  (0.00)2] _ 

(ln(2)]2  +  [ ln ( 3 ) ] 2  +  (ln(2)]2  +  (ln(2)]2  +  [ In ( 1 ) ] 2  + 

+  (ln(l5)]2 


■  0.9867  (acceptable  consistency) 


The  estimate  for  X  is  found  from 

(in  aii)2  -  n  X 


ASL 


i*=l 


n(n-l) 


0.0069 


X  ■  n  +  (n-1)  [exp  (  )  -1]  *  4.0104 

CHECK:  we  should  have  AW  *>  Xw 


12  3  2 

0.4231 

1.6993* 

1.6968 

1/2  12  1 

0.2274 

m 

0.9108 

/s 

;  Xw  ■  4.0104w  «• 

0.9120 

1/3  1/2  J.  1/2 

0.1222 

0.4906 

0.4901 

1/2  1  2  1_ 

0.2274 

0.9108 

0.9120 

a 

The  slight  inconsistency  was  introduced  through  the  13  element  that 
should  have  been  equal  to  4  Instead  of  3,  using  the  correct  value  we  find 


CIS 


t 


v 


.-v* 


x{  =  h[ ln(l)  +  ln(2)  +  In (4)  +  ln(2)]  =  0.6931  > 

*2  ~  0.00  (unchanged) 

=  ‘-sUnOt)  +  ln(4)  +  ln(  1)  +  ln(4)l  =  -0.6931  ; 
x^  =  0.00  (unchanged) 

and  the  normalized  weight  vector  is  given  by 
w1  =  [0.4444,  0.2222,  0.1111,  0.2222} 


wx  =  2.0000 

w2  =  0.00 

«  0.5000 

w.  =  1.00 
4 


The  multiple  correlation  coefficient  for  this  consistent  case  is  equal 
2 

to  R  =1,  and  the  largest  eigenvalue  is  found  to  be  equal  to  X  =  4.00 
Sensitivity  of  the  Priority  Vector 

A  question  that  keeps  coming  up  quite  often  is  the  following:  "how  large 
are  the  perturbations  in  the  eigenvector  if  the  comparison  matrix  is  perturbed 
by  a  certain  amount?"  We  can  show  that  if  a  reciprocal  matrix  A,  is  perturbed 
by  a  reciprocal  matrix  P,  usine  elementwise  (Hadamard)  multiolication  (denoted) 
by  A°P) ,  then  the  new  matrix  A*  is  reciprocal,  too.  The  value  of  perturbation. 
Aw,  in  the  original  priority  vector,  w,  is  found  from 


Aw 


(<y ,w> 


1)  »w 


where 


<y,w>is  the  inner  product  of  y  and  w. 

1  -  is  the  unity  vector 

y  -  is  the  eigenvector  of  the  perturbation  matrix 
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P  -  the  perturbation  matrix  is  obtained  by  taking  elementwise  divisions 
of  elements  of  the  perturbed  matrix  into  those  of  the  original  matrix. 

An  example  will  demonstrate  the  use  of  this  formula. 

Example :  Consider  the  following  comparison  matrix 

1  2  3  A 

1/21  22 
1/3  1/2  1  1 

1/A  1/2  1/2  1 

The  priority  (eigenvector)  vector  of  this  matrix  is  given  by 

w  T  =  [0.4747,  0.2551,  0.1630,  0.1072] 
o 

with  consistency  ratio  equal  to  CR  =  0.017. 

One  may  be  interested  in  finding  out  what  will  r"  e  changes  be  in  the  eigen 

vector  if, instead  of  A  ,we  consider  A,  given  by 

o  i 

12  3  4 

1/2  1  1.5  2 

1/3  2/3  1  4/3 

1/4  1/2  3/4  1 

The  new  priority  vector  is  given  by 

w  T  =  [0.4800,  0.2400,  0.1600,  01200]  ,  CR  -  0.00 

Let's  use  the  formula  to  obtain  this  new  eigenvector.  The  matrix  P  is  the 
matrix  that  when  we  multiply  it  elementwise  (Hadamard)  with  the  matrix 


we  obtain  the  matrix  A^, 


i.e.  , 


1  1  1 
1  0.75  1 

4/3  1  2/3 

1  3/2  1 

The  eigenvector  of  this  matrix  is  given  by 
yT  =  [0.2495,  0.2335,  0.2422,  0.2761] 


We  want  to  demonstrate  the  use  of  the  sensitivity  formula,  i.e.,  derive 


w  by  using  knowledge  of  and  y.  The  formula  is  given  by 


-1 


Aw  =  (<y,  w  >  y  -  1)  ow 
o  o 


and 


w,  =  w  +  Aw 
1  o 


-1 


<y,w  >  =  0.2471  — >  <y,w  >  y  = 


1.0098 

0.9450 

0.9802 

1.1175 


A 

=  y. 


Aw=(y-l)ow  = 
o  ~  o 


0.0098 

0.4747 

0.0047 

-0.0550 

0.2551 

-0.0140 

"0.0198 

o 

0.1630 

-0.0032 

0.1175 

0.1072 

0.0126 

„ 

— 

_  — 

w  + 

o 


0.4747 

0.0047 

0.255] 

-0.0140 

0.1630 

+ 

-0.0032 

ts 

0.1072 

0.0126 
»  - 

L- 
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The  formula  demonstrated  above  relates  changes  In  the  comparison 
matrix  to  changes  in  the  vector  of  local  priorities;  can  we  relate  these 
changes  to  the  vector  of  global  priorities  too? 

Let 

p(l)  =  the  global  priority  vector  at  the  i-th  level 

p(i-l)  =  the  global  priority  vector  at  the  (i-l)-th  level 

*  the  local  priority  vector  relating  all  elements  in  the  i-th 
level,  to  the  j-th  element  in  the  (i-l)-th  level. 

Then  we  have 

A 

p(i)  =  [w1  w2  |  . . .  |wi  |. .  wn]p(l-l)  -  WjPCi-l) 


where  VL  is  the  matrix  whose  columns  are  the  local  priority  vectors 
relating  i  to  level  i-1. 

Change  in  a  comparison  matrix  results  in  a  change.  Aw,  in  the  local 
priority  vector  and,  therefore,  will  change  also  the  global  priority  vector. 


Let 

A 

w . 
3 


new  local  priority  vector  resulting  from  changes  in 

the  comparison  matrix  relating  elements  in  the  i-th  level  to  the 

j-th  element  in  the  previous  level. 


Then 


A 
W  . 


AWj 


Now  we  can  define 

W  -  [WjJ  w2  |  ...  |  v  |...  Wn)  «  W  +  AW 
where 

AW  «  (Aw1j  A«2  I  •  •  •  I  i«nl 

and  all  the  columns  of  this  matrix  are  obtained  from  the  formula.  Then, 
the  new  global  priority  vector,  p(i),  is  given  by 

P(i)  ■  w  p(i-l)  ■  (w  +  AW)p(l-i)  *  p(i)  +  AWp(i-l). 
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Concordance  Coefficients 


The  concept  of  concordance  is  designed  to  help  determine  the  overall 
attractiveness  of  an  alternative  course  of  action.  This  attractiveness  depends 
not  only  on  the  priority  assigned  to  this  alternative  but  also  whether  it 
addresses  all  criteria  or  just  those  with  the  highest  priority.  The  concept 
of  concordance  is  very  useful  for  the  sensitivity  analysis  associated  with  ^;-,c 
options  generation  process.  This  is  so  because  it  allows  us  to  identify 
whether  an  alternative  comes  out  ahead  of  the  other  (on  a  priority  scale) 
just  because  it  addresses  a  high  priority  objective  and  then,  if  this  object¬ 
ive  becomes  less  important  later  on  (or  there  was  an  error  in  the  thinking 
process  that  resulted  in  it  having  such  a  high  score),  it  may  become  of  little 
value  because  it  cannot  contribute  to  the  other  objectives  that  may  still  hold 
valid . 


Global  priorities  are  obtained  as  a  weighted  average  of  local  pr:o-ities, 
where  the  weighting  factors  are  the  global  priorities  of  the  elements  in  the 
previous  level.  We  will  define  now  the  global  priorities  to  be  a  solution 
of  a  minimization  problem.  The  global  priorities  are  those  that  minimize  the 
sum  of  weighted  squared  deviations  between  the  local  priorities  and  the  global 
priorities.  The  weighting  factors  are  the  global  priorities  of  the  elements 
in  the  previous  level.  That  is,  the  problem  is  described  as  follows: 

Min  ]>  =  Min 

w .  i  w . 

1  1 


y  P,  (w  -  w. ) 
k  lk  1 


where 


P,  - 


lk 


global  priority  of  kt!l  element  in  the  previous  level. 

global  priority  of  iC^  element  in  the  level  under  consideration 

-  local  priority  of  element  i  with  respect  to  element  k  in  the 

previous  level. 
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The  optimal  solution  for  w^  is  obtained  from 


3w.  0  =  “2  X  Pk  (wik  "  “i5 

1  k=l 


-  -2l‘Pl“il  +  P2“12  +  +  Pn“ln>  -  "i(Pl  +  P2  +  +  P„>: 


but  since,  for  every  level,  we  have 


V  p.  =  l 
k-1  k 


we  find  the  optimal  global  priority  to  be  given  by 


*  r- 

w.  =  >  P  w 

i  f  k  lk 
k 

which  also  happens  to  be  the  way  we  computed  the  global  priorities,  i.e.,  the 
formulation  of  the  optimization  problem  did  not  change  the  way, we  compute 
these  global  priorities.  What  have  we  gained? 

Before  we  know  the  local  priorities,  w^,  each  element  in  the  current 
level  is  equally  attractive,  i.e.,  its  prior  global  priority  is  (1/n)  and 
the  value  of  G  in  that  case  is  =  \/n  (this  acts  as  a  null  hypothesis). 
When  the  local  priorities,  wik  are  known,  the  global  priority,  w^  of  the 
ith  element  can  be  computed;  this  reduces  the  value  of  G^  by  (w^^  -  1/n)  . 

A  concordance  coefficient  (analogous  to  a  multiple  regression  coefficient) 
can  now  be  defined  by 

I  w^  -  1/n 


1  [2  pk(vik  -  1/n)2] 


- 

The  square  of  this  coefficient  provides  the  relative  reduction  in  G^. 

The  coefficient  of  concordance  is  positive  for  elements  with  global 
priority  higher  than  average,  zero  for  average  elements,  and  negative  for 
below  average  elements.  Elements  with  great  variance  in  their  local  prior¬ 
ities  will  have  low  concordance  coefficients  (variances  in  local  priorities 
affect  the  denominator  in  the  concordance  expression). 

Using  the  global  priority  and  the  concordance  coefficient  for  an  alter¬ 
native  (or  an  element  in  general)  one  can  arrive  at  several  classifications 
of  alternatives.  These  classifications  are  not  unique  but  the  following 
guidelines  can  be  used.  Priorities  which  are  20%,  or  more,  higher  than  the 
average  priority,  1/n,  are  considered  high.  Priorities  below  80%  of  average 
are  considered  low.  Concordance  coefficient  greater  than  0.50  and  less  than 
-0.50  are  considered  high. 

Note  that  no  alternatives  wil^.  ever  be  found  in  the  shaded  regions,  since 
whenever  a  global  priority  drops  below  average  it  has  a  negative  concordance 
coefficient . 

Final  Remark:  The  coefficient  of  concordance  serves  as  a  rough  indication  of 
the  sensitivity  of  the  overall  priority  with  respect  to  changes  in  its  assess¬ 
ments,  or  to  changes  in  global  priorities  of  elements  in  the  preceding  level. 

Example :  Consider  the  following  hierarchical  decision  problem 


C22 


where  the  following  comparison  matrices  are  given 


T 

A 

B 

C 

A 

1 

2 

3 

B 

1/2 

1 

2 

C 

1/3 

1/2 

1 

A 

!  D 

E 

F 

D 

i 

7 

6 

E 

1/7 

1 

2/3 

F 

1/6 

1.5 

1 

similar  way  we 

B 

D 

E 

F 

D 

1 

1/6 

1/5 

E 

6 

1 

2 

F 

5 

1/2 

1 

C 

D 

E 

F 

D 

1 

1/8 

l/< 

E 

8 

1 

2 

F 

6 

1/2 

1 

=  0.60 

=  1.82 

=  0.00 

k 

=  1.00 

=  0.60 

(°3 

=  0.55 

=  1.25 

( 

=  3.40 

=  -0.78 

* 

{  W2 

=  0.46 

=  -0,46 

I 

=  0.63 

V  3 

(0.08\ 
0.58  , 
0.34/ 


/  0.06  \ 

WC  'I0'59  J 
\0. 34  / 


and  global  priorities  are  given  by 


The  concordance  coefficients  are  found  from 


From  this  we  see  that  only  alternative  D  falls  in  an  acceptable  region 
while  both  E  and  F  are  unacceptable  (E  is  better  than  F,  though). 


APPENDIX  D:  STIMULUS  MATERIAL  FOR  THE  STANFORD  EXPERIMENT 


This  appendix  contains  copies  of  a  selection  of  the  stimulus  material 
used  by  the  subjects  in  the  Stanford  Experiment.  It  is  self  explanatory 
and  includes  the  basic  introductory  material  received  by  all  subjects, 
the  Simple  Environment  and  the  Post-Test  Questionnaire. 


EXPERIMENTAL  DESCRIPTION  PROTOCOL 


Good  laf™rnoon]*  t*ian'c  y°u  ^or  agreeing  to  participate  in  this 
experiment.  We  are  trying  to  devise  ways  of  helping  decision  makers 
in  the  generation  of  creative  solutions  to  complex  decision  problems, 
and  the  purpose  of  the  experiment  today  is  to  test  the  effectiveness 
of  one  of  the  aids  that  will  be  provided. 

Both  before  you  begin,  and  when  you  finish,  we  will  ask  you  to 
fill  out  a  simple  questionnaire,  that  is  designed  to  provide  us 
with  some  attitudinal  information. 

The  experimental  session  proper  will  start  when  you  are  given 
a  description  of  the  problem  and  an  introduction  to  the  aids.  The 
session  can  last  up  to  two  hours,  but  you  may  stop  whenever  you  feel 
you  have  completed  the  task.  One  of  us  will  be  present  at  all  times 
and  will  answer  any  questions  concerning  the  experimental  procedures. 

Remember  that  this  is  not  a  test  of  your  ability  but  is  simply 
a  way  of  helping  us  define  some  useful  decision  aids. 


Pre-Test  Questionnaire 


The  prupose  of  this  questionnaire  is  to  establish  some  basic  biograph¬ 
ical  data,  and  to  determine  your  prior  knowledge  about  the  problem  you  will 
be  asked  to  work  on.  We  hope  to  be  able  to  correlate  this  information  with 
the  results  we  obtain  from  our  experiments. 

Please  answer  all  the  questions.  When  you  have  finished,  return  the 
questionnaire  to  the  experimenter.  Thank  you. 


Name : 


Age:  Sex:  Male _  Female _ 

Status  at  Stanford:  (circle)  1st  year  2  34  graduate 

Major  area  of  study: 

Your  home  town/state: 

Do  you  know  of  the  Med-Fly  infestation  in  the  Bay  Area?  Yes _  No _ 

If  yes,  then  how  would  you  describe  your  knowledge  of  the  problem  and  attempts 
to  solve  it? 

Marginal  Very  Detailed 


Have  you  heard  that  there  have  been  attacks  on  women  on  the  Stanford  Campus 
over  the  last  six  months?  Yes _  No _ 

If  yes,  then  how  would  you  describe  your  knowledge  of  the  problem  and  attempts 
to  solve  it? 

Marginal  Very  Detailed 
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V  -  V 


THANK  YOU  for  agreeing  to  participate  in  this  experiment.  Should  you  have 
any  questions,  a  member  of  our  team  will  be  present  in  the  room  at  all  times. 
Take  as  much  time  as  you  need  to  complete  the  experiment.  Note  that  this  is  not 
a  test  of  your  personal  ability. 

Please  begin  by  reading  the  OVERVIEW  on  the  following  page. 
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OVERVIEW 


Wc  arc  interested  in  aiding  individuals  in  generating  alternatives  (options) 
to  solve  a  problem  (achieve  a  goal).  This  is  especially  important  in  stressful 
situations,  where  people  typically  generate  a  very  limited  number  of  solutions. 

Our  proposed  "aid"  is  a  package  of  tools  including: 

INFORMATION  -  A  selection  of  information  that  may  be  relevant 

to  the  problem. 

STRUCTURING  AIDS  -  Tools  to  help  organize  the  basic  elements  of  the 

problem. 

CREATIVITY  AIDS  -  Tools  designed  to  stimulate  creative  thinking. 

After  completing  a  few  preliminary  tasks,  you  will  be  presented  will  three 
envelopes,  each  containing  one  of  the  above  set  of  tools.  You  may  use  all,  some 
or  none  of  the  aids,  and  in  any  order.  You  may  regard  the  use  of  the  aids  much 
like  you  would  approach  a  buffet:  there  are  appetizers,  entrees  and  desserts  to 
select  from.  You  may  eat  anything  you  wish,  in  any  amount  and  in  any  order. 

The  goal  of  this  experiment  is  to  generate  options  for  the  problem  posed  on 
the  following  page.  Remember  that  there  are  no  right  or  wrong  answers. 

Now  please  read  the  problem. 
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CAMPUS  SECURITY  SCENARIO 


Since  October,  1981,  there  have  been  a  number  of  assaults  and 
rapes  on  the  Stanford  campus.  The  first  attack  occurred  in  early 
October,  at  8:30  p.m.  near  Tresidder  Memorial  Union.  On  November  5, 
at  10:30  a.m.,  a  woman  was  attacked  while  jogging  near  the  stadium. 
Several  days  later,  a  third  woman  was  attacked  at  5:45  p.m.,  as  she 
walked  between  Memorial  Church  and  building  50.  Two  men  were  believed 
to  have  been  involved  in  the  latter  case,  but  the  victim's  eyes  were 
covered,  and  no  description  was  possible.  In  mid-December,  at  about 
6  p.m.,  a  woman  was  accosted  in  the  parking  lot  near  the  Physics 
Tank,  and  threatened  with  kidnap.  The  latest  incident  took  place  on 
January  5,  when  a  woman  was  assaulted  shortly  after  1:00  p.m.  in 
the  Arobretum  near  the  Stanford  Mausoleum. 

The  Stanford  Police  Department  increased  both  uniformed  and  plain¬ 
clothes  patrols.  The  Santa  Clara  County  Sheriff's  Department  increased 
coverage  of  the  campus  by  patrol  units.  The  A.S.S.U.  bought  and  gave 
out  several  thousand  whistles,  with  the  instructions  to  blow  them  if 
you  are  attacked  or  hear  someone  in  trouble,  and  to  move  toward  the 
whistle  if  you  hear  it.  The  University  now  allows  parking  in  A  lots 

after  3:00  p.m.,  so  that  women  may  move  their  cars  closer  to  the 

buildings  before  evening. 

Consider  yourself  a  member  of  a  special  advisory  panel  set  up  by 
the  President's  office.  Your  task  is  to  generate  as  many  solution  or 
partial  remedies  to  the  campus  security  problem  The  solutions  may  apply 
at  any  level — actions  individuals  may  take,  or  the  University,  the 
Stanford  Police  Department,  or  any  level  at  which  you  can  generate 
an  option.  Aim  for  as  many  ideas  as  you  can,  even  if  they  don't 
initially  appear  feasible.  Other  members  of  the  advisory  panel  may 
be  able  to  use  your  idea  as  a  jumping  off  point.  Just  try  to  come 

up  with  as  many  possible  approaches  to  the  problem  as  you  can. 
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Now  that  you  have  read  the  scenario,  please  describe  the  problem  in  your 


own  words  on  the  sheet  labeled  PROBT.EM  DESCRIPTION. 


-  turn  to  next  page  - 


At  this  time,  please  record  any  initial  solutions  (options)  you  may  have 


the  sheet  labeled  INITIAL  OPTIONS. 


-  turn  to  next  page  - 
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Inform  the  member  of  our  team  present  that  you  are  now  ready  to  utilize 


the  aids  to  try  to  generate  more  options  for  the  problem  you  read  earlier. 

You  will  find  more  detailed  information  about  each  set  of  tools  on  the  face 
of  the  envelopes. 

Please  record  any  new  options  on  the  sheet  labeled  NEW  OPTIONS.  When  you 
feel  that  you  have  completed  the  experiment,  please  notify  the  member  of  our 
team  present. 

Thanks  again! 
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STRUCTURING  AIDS 


In  this  package  you  will  find  two  tools  designed  to  help  you  think  about 
the  structure  of  the  problem. 

1.  A  simple  checklist 

2.  A  structure  template 

The  first  of  these  is  very  straightforward  and  is  really  just  a  way  of  reminding 
you  of  various  aspects  of  the  problem.  The  second  is  more  elaborate  and  requires 
you  to  make  a  conscious  explicit  effort  to  record  the  main  elements  of  the 
problem. 

Use  these  aids  only  if  you  need  to.  View  them  as  tools  for  problem  solving, 
and  not  as  tasks  that  must  be  completed. 


CHECKLIST  STRUCTURING  AID 

In  thinking  through  the  campus  security  problem,  you  may  want  to 
consider  the  following  aspects: 

ECONOMIC 

ENVIRONMENTAL 

POLITICAL 

SOCIAL 
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Structuring  Template 


The  purpose  of  the  aid  is  to  help  you  organize  the  important  elements  of  the 
problem.  Going  through  this  exercise  should  improve  your  understanding,  and  help  you 
generate  more  options. 

On  the  attached  sheet  you  will  find  a  series  of  empty  boxes  with  labels  at  the 
left  hand  edge.  The  idea  is  to  enter  a  brief  description  of  the  labeled  aspect  of 
the  problem.  So,  in  one  box  at  the  Actors  level  you  could  write  "students" 

Use  as  many  boxes  as  you  need  at  each  level,  and  add  more  boxes  if  necessary.  By 

writing  down  as  many  actors people,  groups,  or  even  nonhumans - you  may  be  able  to 

consider  the  problem  from  their  perspective,  and  so  generate  more  options  than 
if  you  just  consider  the  problem  from  one  perspective.  Also,  you  may  stumble  onto 
the  resources  of  a  group  which  you  totally  forgot  to  consider.  You  may  ste  how 
to  satisfy  the  goals  of  one  subset  of  the  actors,  and  solving  a  subproblem  is  a 
step  to  solving  the  whole  problem.  In  short,  the  structuring  template  is  meant 
to  stretch  your  conception  of  the  problem,  broaden  it  as  much  as  you  can,  and  give 
you  ideas  about  attacking  the  problem  in  different  areas. 

The  structuring  template  is  not  to  be  considered  a  whole  extra  problem.  Use  it 
as  much  or  as  little  as  you  wish.  As  you  fill  it  out,  write  down  any  options  that 
occur  to  you  on  the  sheets  labeled  "new  options". 

An  example  of  a  template  filled  out  by  someone  working  on  the  problem  of 
medflies  in  the  Bay  area  is  provided  as  a  guide. 
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Structuring  Template 


CREATIVITY  AIDS 


In  this  package  you  will  find  three  tools  designed  to  stimulate  your 
creativity 


1.  Challenge  Assumption  Protocol 

2.  Free  Associations 

3.  Brainstorm  Questions. 

All  are  straightforward  and  selfexplanatory,  although  #1  will  probably 
require  most  effort  to  use. 

Do  not  feel  obliged  to  use  these  tools.  Work  with  them  only  when  and 
if  you  want  to.  Remember  that  they  are  aids  for  problem  solving,  not  problems 
themselves . 


m 


FREE  ASSOCIATIONS 


Sometimes  it's  useful  to  take  a  break  from  a  problem  and  think  in 
one  entirely  different  area.  To  help  you  do  so,  we  will  give  you  a 
series  of  cue  words.  Read  one  cue  word,  and  to  respond  with  the  first 
word  which  comes  into  your  mind.  Write  it  in  the  blank  next  to  the 
cue  word.  Then  go  on  to  the  next  cue  word.  Continue  for  all  of  the 
cue  words. 
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CUE  WORDS 


RESPONSE 


1. 

2. 


3. 


4. 

5. 


6. 

7. 

8. 

9. 

10. 

11. 

12. 

13. 

14. 


rabbit 

conceal 

orbit 

kitchen 

fragment 

hinge 

kiwi 

petals 

electric 

random 

ski 

horoscope 

dancing 

antibody 
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proud 


BRAINSTORM  QUESTIONS 


One  way  of  creating  a  new  view  of  the  problem  is  to  ask  yourself 
thought-provoking  questions  such  as  those  that  appear  on  the  following 
page. 


If,  in  thinking  through  these  questions,  new  options,  or  modifica¬ 
tions  to  existing  ones,  occur  to  you  then  please  enter  them  on  the 
worksheet  labelled  "New  Options".  Feel  free  to  spend  as  much  time  as 
you  need  on  this  process. 


019 


Have  I  Thought-up  all  the  phasea  of  the  problem? 

Which  aub-problems  aeem  moat  important? 

Ia  there  data  that  might  help? 

Where  might  thia  data  be  found? 

What  ideaa  do  I  have  a o  far? 

How  would  I  begin  to  te8t  the  validity  of  the  optiona 
I've  generated  ao  far? 

Have  I  imagined  all  poaaible  contingencies? 
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CHALLENGE  ASSUMPTIONS  PROTOCOL 


A  good  way  of  generating  insights  into  a  problem  is  to  make  a  list  of  your 
assumptions,  and  then  challenge  it. 

Start  by  briefly  stating  your  assumptions  on  the  sheet  labeled  ASSUMPTIONS. 
Please  give  this  task  careful  thought,  as  many  times  we  are  not  consciously  aware 
that  we  have  made  assumptions.  For  example,  if  you  heard  the  word  "newlyweds",  you 
are  likely  to  be  inclined  to  envision  a  young  man  and  woman  when  in  fact,  the 
newlyweds  could  be  senior  citizens. 


After  recording  your  assumptions  on  the  left  side  of  the  sheet,  choose  the 
most  restrictive  one  in  your  estimation,  and  examine  the  consequences  of  assuming 
it  is  not  true.  To  help  you  do  that,  try  to  write  down  its  negation  on  the  right 
side  of  -he  paper,  and  do  so  in  a  positive  way.  For  example,  instead  of  writing: 


NEGATED  ASSUMPTION 
no  gravity 

write: 


ASSUMPTION 

gravity 


ASSUMPTION 


NEGATED  ASSUMPTION 


gravity 


people  float 


Try  to  use  the  negated  assumption,  however  implausible  it  may  seem. 

Continue  to  do  this  for  all  assumptions. 

If  at  any  time  you  find  that  new  options,  or  modifications  to  existing  ones, 
occur  to  you,  please  remember  to  include  them  on  the  sheet  labeled  NEW  OPTIONS. 
Feel  free  to  spend  os  much  time  as  you  need  on  this  process. 
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ASSUMPTIONS 


ASSUMPTION 


NEGATED  ASSUMPTION 


1. 


1. 


2. 


2. 


3. 


3. 
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Post-Test  Questionnaire 


The  purpose  of  this  questionnaire  is  to  get  your  opinion  about  the 
experimental  procedures  you  have  just  experienced.  Please  be  as  frank 
and  honest  as  possible  in  responding  to  these  questions.  Your  answers 
wi] 1  be  entirely  confidential  and  will  be  used  only  as  supporting  anecdotal 
evidence  in  any  conclusions  we  reach  as  a  result  of  this  experiment. 


■>  :  i  :  v* 
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Ql.  Do  you  think  that  the  overall  procedure  helped  you  generate  more 

solutions  than  you  might  have  if  left  on  your  own?  Please  explain. 


Q2.  What  were  your  general  impressions  of  the  experimental  procedure? 


Q5.  Can  you  think  of  any  tools  or  aids  you  would  have  found  useful,  but 
which  were  not  provided?  If  so,  please  explain. 


Q6.  Are  there  any  other  remarks  you  would  like  to  address  to  us? 
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MISSION 

of 

Rome  Air  Development  Center 

RAVC  plan*  and  execute*  re&earch,  de.veZopme.nt,  te&t  and 
6  elected  acquisition  pn.ogn.am6  Zn  iupport  of  Command,  Contnol 
Communication*  and  Intelligence  (C3 1)  activities.  Technical 
and  engineenZng  *upport  uxithin  area*  of  technicai  competence 
i6  provided  to  ESP  Program  Office*  ( P06 )  and  other  ESP 
eZementi .  The  principal  technical  mission  area&  are 
communication*,  electromagnetic  guidance  and  control,  *ur- 
veitiance  of  ground  and  aero6pace  object*,  intelligence  data 
collection  arid  handling,  information  6y6tem  technology, 
iono6pheric  propagation,  solid  6 tote  science*,  microooave 
physic*  and  electronic  reliability,  maintainability  and 
compatibility. 


